Can wget Read URLs From Standard Input?

The standard command-line utility wget is widely used for downloading files from the internet, but it cannot read URLs directly from standard input (stdin) by default without a specific command modifier. This article provides a quick overview of how to pipe URLs into wget, demonstrating the use of the -i - option to enable standard input reading, exploring alternative solutions like xargs, and comparing wget with curl for stream-based downloading.

The Default Behavior of wget

When you pass data to wget via a pipe (|), the utility normally ignores the standard input and expects URLs to be provided as explicit command-line arguments. For example, running a command like echo "https://example.com/file.txt" | wget will result in an error or cause wget to display its help menu because it does not see any target destinations in the arguments.

How to Make wget Read from Stdin

To force wget to listen to standard input, you must use the -i (or --input-file) option followed by a hyphen (-). The hyphen acts as a special filename that represents standard input in Unix-like environments.

The Standard Syntax

echo "https://example.com/file.txt" | wget -i -

Processing Multiple URLs

If you have a command or script that outputs a list of multiple URLs, the same structure applies. wget will process each URL line by line:

cat url_list.txt | grep "downloads" | wget -i -

Alternative Approach Using xargs

Another common way to achieve this behavior without the -i - flag is by using xargs. The xargs command takes items from standard input and converts them into arguments for the command that follows it.

echo "https://example.com/file.txt" | xargs wget

While this works perfectly for a small number of links, using wget -i - is generally preferred for massive lists, as xargs can sometimes hit system command-line length limits (ARG_MAX).

wget vs. curl for Standard Input

If your workflow relies heavily on piping data and standard streams, it is worth noting the behavioral difference between wget and curl.

While wget is designed to save downloaded content directly to files on your disk, curl outputs the downloaded content directly to standard output, making curl a more natural fit for complex command-line pipelines where the data needs to be processed further before saving.