Can wget Send a Specific Referer URL?

The wget command-line utility fully supports sending a specific Referer URL in its HTTP requests to mimic organic web traffic or bypass basic hotlinking restrictions. This article provides a quick overview of how to use the --referer option, demonstrates practical command-line examples, and explains how to combine it with user-agent strings for more robust web scraping and automated downloads.

Using the –referer Option in Wget

By default, wget does not include a Referer header in its HTTP requests. However, many web servers inspect this header to ensure that requests originate from their own site or trusted third parties, often to prevent hotlinking of images and media. You can easily forge or specify this header using the --referer flag (or its short form, -E on some older systems, though --referer is universally preferred).

The basic syntax for the command is:

wget --referer="https://example.com/source-page" "https://example.com/target-file.zip"

Practical Examples

Here are a few common scenarios where modifying the Referer header is necessary:

Bypassing Hotlink Protection: If a website blocks direct downloads of an image unless you are viewing the page it is hosted on, you can pass that exact page URL as the referer. wget --referer="https://blog.mysite.com/article1" "https://blog.mysite.com/images/photo.jpg"
Simulating Search Engine Traffic: Some documentation sites or forums only allow access if you arrive via a search engine. You can simulate a referral from Google like this: wget --referer="https://www.google.com/" "https://example.com/restricted-tuts"

Combining Referer with User-Agent

Many server-side security systems look at both the Referer and the User-Agent headers to detect automated bots. Because the default wget user-agent identifies itself clearly as Wget/version, it is often wise to change both headers simultaneously to mimic a standard desktop browser.

wget --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64)" \
     --referer="https://www.google.com/" \
     "https://example.com/downloads/data.pdf"

Using these options gives you precise control over the metadata sent by wget, making it a highly effective tool for interacting with restrictive web servers.