How to Set a Delay Between wget Downloads?

When scraping websites or downloading large batches of files using the wget command-line utility, sending rapid-fire requests can overwhelm the target server or get your IP address temporarily banned. To prevent this, wget provides built-in options to configure a specific wait time between file retrievals. This article covers how to implement a fixed delay, how to introduce randomized wait times to mimic human behavior, and how to apply these settings efficiently across multiple URLs.

Using the Wait Option for a Fixed Delay

The primary mechanism for pausing between downloads in wget is the --wait option (or its short form, -w). By appending this option to your command, you instruct the utility to pause for a designated number of seconds after completing each file retrieval before initiating the next one.

wget --wait=5 http://example.com/file1.zip http://example.com/file2.zip

In this example, wget will download file1.zip, wait exactly 5 seconds, and then proceed to download file2.zip.

Introducing Randomization to Mimic Human Behavior

If you are downloading a large number of files from a server with strict anti-scraping policies, a predictable, rigid delay might still trigger security alerts. To make your automated traffic look more natural, you can combine the wait command with the --random-wait flag.

wget --wait=10 --random-wait -i download_list.txt

When --random-wait is enabled, wget will vary the pause duration linearly between 0.5 and 1.5 times the value you specified in --wait. In the command above, instead of waiting exactly 10 seconds every time, the tool will dynamically calculate a random pause between 5 and 15 seconds after each download.

Managing Large Batches via Input Files

For scenarios involving dozens or hundreds of links, pasting URLs directly into the terminal is impractical. The most efficient workflow involves saving your target URLs into a plain text file (e.g., urls.txt), with one link per line, and referencing it using the -i option alongside your delay settings.

By combining these parameters, you can execute a command like wget -w 4 --random-wait -i urls.txt. This structure ensures your automated download tasks run smoothly in the background while respecting the bandwidth and security thresholds of the hosting server.