How Does wget Handle Slow Connections?

This article explores how the wget command-line utility manages file downloads over slow or unstable network connections. It covers the tool’s built-in mechanisms for automatic retries, resuming interrupted transfers, and managing timeouts. Additionally, we will look at specific command-line flags you can use to optimize wget for unreliable networks, ensuring successful data delivery even under poor connectivity conditions.

Automatic Retries and Connection Timeouts

The primary strength of wget in unstable network environments is its aggressive, automated retry logic. When a connection drops or times out, wget does not simply give up and exit. Instead, it enters a loop to re-establish the connection and pull the remaining data.

By default, wget will attempt to retry a download up to 20 times before declaring a failure. It also employs a standard timeout period (usually 900 seconds or 15 minutes of silence from the server) before recognizing that a connection has stalled and triggering a retry.

Resuming Interrupted Downloads

When a network connection completely breaks mid-download, restarting the file transfer from scratch is inefficient and time-consuming. wget solves this with its continuation feature.

Using the -c or --continue flag, wget checks the size of the partially downloaded local file and sends a specific request to the server to stream only the remaining bytes. This capability relies on the remote server supporting “Range” requests, a standard feature in most modern web servers.

Key Configuration Flags for Unstable Networks

While wget has robust default settings, you can fine-tune its behavior using specific command-line options to better survive highly volatile connections:

• --tries=number: Changes the number of retry attempts. Setting --tries=0 or --tries=inf forces wget to retry infinitely until the file is successfully downloaded.
• --timeout=seconds: Lowers the time wget waits for a response before giving up on a stalled connection. For highly unstable networks, reducing this to 30 or 60 seconds forces quicker reconnection attempts.
• --wait=seconds: Introduces a pause between retry attempts, which prevents overwhelming a struggling server or network node.
• --random-wait: Varies the wait time between retries to avoid triggering anti-bot or rate-limiting defenses on the host server during frequent reconnections.

Handling Total Bandwidth Constraints

On slow connections where saving bandwidth for other applications is necessary, wget offers rate-limiting features. The --limit-rate flag allows you to cap the download speed (e.g., --limit-rate=50k). Limiting the speed can actually stabilize a connection on lower-tier networks by preventing the download from fully saturating the available bandwidth, which often leads to packet loss and dropped connections.