How wget Downloads Files in Linux

The wget utility is a non-interactive, command-line tool used in Linux to retrieve files over networks using protocols such as HTTP, HTTPS, and FTP. This article explains the end-to-end mechanism of how wget operates, detailing the process from the moment a user enters a command to the final writing of data onto the local storage drive.

1. URL Parsing and Protocol Selection

When a user issues a command like wget https://example.com/file.zip, the utility first parses the provided URL string. It breaks the URL into distinct components: the protocol scheme (http, https, or ftp), the host domain name (example.com), the optional port number (defaulting to 80 for HTTP or 443 for HTTPS), and the remote file path (/file.zip). Based on the scheme, wget configures the appropriate network protocol handlers internally.

2. DNS Resolution

Before connecting to the server, wget must resolve the human-readable domain name into an IP address. It calls system-level resolver libraries (typically via the getaddrinfo function in standard C libraries). This queries the configured Domain Name System (DNS) servers to retrieve either an IPv4 (A record) or IPv6 (AAAA record) address for the destination host.

3. Establishing the Network Connection

Once the IP address is determined, wget creates a network socket and initiates a standard TCP three-way handshake (SYN, SYN-ACK, ACK) to establish a reliable connection with the target server on the designated port.

If the protocol is HTTPS, wget initiates a Transport Layer Security (TLS/SSL) handshake immediately following the TCP connection. During this phase, it validates the remote server’s SSL certificate against the local system's trusted Certificate Authority (CA) bundle and negotiates encryption keys using libraries like OpenSSL or GnuTLS.

4. Sending the HTTP Request

With a secure connection established, wget formats and transmits an HTTP GET request over the socket. The request includes essential headers such as:

• Host: Specifies the target domain.
• User-Agent: Identifies wget and its version.
• Accept: Informs the server what media types are acceptable.
• Range: Sent if the download is resuming an interrupted file (using the -c flag) to request only the remaining bytes.

5. Evaluating the Server Response

The remote server processes the request and responds with an HTTP status code alongside response headers. wget evaluates this response:

• 200 OK: The request succeeded, and data transmission begins.
• 206 Partial Content: Returned when resuming a download via byte ranges.
• 3xx Redirection: If the resource has moved (e.g., 301 or 302), wget reads the Location header and automatically repeats the connection process for the new address.
• 4xx/5xx Client/Server Errors: wget terminates or executes retry logic based on its configuration flags (e.g., --tries).

6. Streaming Data and Writing to Disk

Once data delivery begins, wget allocates an internal memory buffer to read incoming network packets from the socket.

As binary chunks are read into memory, wget makes write() system calls to stream the data directly to a file on the local filesystem. By default, the local file is named after the last component of the URL path unless specified otherwise with the -O option. During this phase, wget calculates transfer speeds, estimates time of arrival (ETA), and renders a real-time progress bar to the terminal standard error (stderr).

7. Connection Termination

After the final byte is transferred (signaled either by reaching the byte count indicated in the Content-Length header or by a closed connection in chunked transfers), wget closes the local file descriptor. It then sends a TCP FIN packet to gracefully terminate the network connection, logs a completion message, and exits with a return code of zero.