How to Download a List of URLs From a File Using Wget?

Downloading multiple files simultaneously can be a tedious chore if done manually, but the wget command-line utility offers a powerful, automated solution. This article provides a quick overview and step-by-step guide on how to read a list of URLs from a plain text file and download them all at once using a single terminal command. We will cover the basic command syntax, how to handle potential download errors, and useful optional flags to customize your download process.

The Basic Command Syntax

The most efficient way to process a list of links with wget is by using the -i (or --input-file) option. This flag tells the utility to read the specified file and download every URL found inside it, line by line.

Before running the command, ensure you have a plain text file (for example, urls.txt) containing one full URL per line:

https://example.com/file1.zip
https://example.com/file2.pdf
https://example.com/file3.jpg

Once your file is ready, open your terminal, navigate to the directory where your text file is located, and execute the following command:

wget -i urls.txt

Managing and Optimizing Your Downloads

When downloading large batches of files, standard configurations might not always suit your network conditions or storage preferences. You can append various flags to the base command to gain better control over the execution.

• Set a Download Directory (-P): By default, files save to your current working directory. To route them to a specific folder, use the prefix flag followed by the target path.

wget -P /path/to/directory -i urls.txt

• Handle Unstable Connections (-c): If a download gets interrupted due to a network drop, adding the continue flag ensures wget picks up right where it left off instead of starting from scratch.

wget -c -i urls.txt

• Prevent Server Overload (-w): Sending too many rapid requests can cause servers to block your IP address. You can introduce a mandatory wait time (in seconds) between each file download.

wget -w 5 -i urls.txt

Troubleshooting Common File Format Issues

A frequent issue when using this method arises from file encoding differences, particularly if the text file was created on a Windows machine but is being read on a Linux or macOS system. Windows uses hidden carriage return characters (\r\n) at the end of lines, which can corrupt the URLs in the eyes of wget.

If your terminal throws errors stating that the URLs cannot be found or contain strange trailing characters, you can clean the text file using the dos2unix utility before running your download:

dos2unix urls.txt
wget -i urls.txt