How to Check URL Validity Using Wget?

This article provides a quick guide on how to use the wget command-line utility to verify if a URL is valid and active without downloading the target file. You will learn the specific flags required to perform a “headline-only” check, look at clear examples of the command in action, and understand how to interpret the terminal output to confirm a URL’s status.

The Core Command

To check if a URL is valid without downloading anything, you need to instruct wget to fetch only the server’s response headers. This is done by combining the --spider and -S (or --server-response) flags.

The standard syntax is:

wget --spider -S https://example.com

Understanding the Flags

• --spider: This puts wget into “web spider” mode. Instead of downloading the pages or files, it simply checks for their existence.
• -S (or --server-response): This tells wget to print the headers sent by the HTTP server, allowing you to see the exact status codes (like 200 OK or 404 Not Found).

Common Examples and Output

1. Checking a Valid URL

When you run the command against a valid, working URL, the server will return a 200 OK status code.

wget --spider -S https://www.google.com

What to look for in the output:

HTTP/1.1 200 OK Remote file exists.

2. Checking a Broken or Invalid URL

If the URL is broken, typed incorrectly, or the page no longer exists, the server will usually return a 404 Not Found status code.

wget --spider -S https://www.google.com/thispageisfake

What to look for in the output:

HTTP/1.1 404 Not Found Remote file does not exist – broken link!!!

Advanced Tip: Shhh, Make it Quiet

If you are using this command inside an automated bash script, the default output of wget can be too noisy. You can combine the spider flag with the quiet flag (-q) and check the exit status of the command instead.

wget --spider -q https://example.com

• If the command exits with 0, the URL is valid.
• If it exits with a non-zero error code (like 8), the URL is invalid or unreachable.