How Does wget Handle HTTP Authentication?

The wget command-line utility handles HTTP Basic and Digest authentication mechanisms by accepting user credentials via command-line flags or configuration files, which it then encodes into the appropriate HTTP headers to authenticate requests. While Basic authentication transmits credentials in an easily reversible Base64-encoded string, Digest authentication applies a MD5 or SHA-256 cryptographic hash function to prevent passwords from being sent in plaintext. This article explores how wget processes both methods, the specific commands required for each, and best practices for securing your credentials during automated file downloads.

Basic Authentication in wget

HTTP Basic authentication is a simple challenge-and-response mechanism where the server requests credentials, and the client responds with a username and password joined by a colon and encoded in Base64. Because Base64 is not encryption, this method relies entirely on HTTPS (SSL/TLS) for security.

When you use wget for Basic authentication, you provide the credentials using the --user and --password flags. Alternatively, you can embed them directly into the URL.

• Using standard flags: wget --user=myusername --password=mypassword https://example.com/protected-file.zip
• Using URL embedding: wget https://myusername:mypassword@example.com/protected-file.zip

Behind the scenes, wget takes the string myusername:mypassword, encodes it into Base64 (e.g., bXl1c2VybmFtZTpteXBhc3N3b3Jk), and transmits it in the request header like this: Authorization: Basic bXl1c2VybmFtZTpteXBhc3N3b3Jk

Digest Authentication in wget

HTTP Digest authentication is a more secure alternative designed to fix the inherent flaws of Basic authentication over unencrypted connections. Instead of sending the password over the network, the server sends a unique, temporary challenge string called a nonce (number used once). The client then hashes the username, password, nonce, HTTP method, and requested URI together to create a unique response string.

wget supports Digest authentication automatically out of the box using the exact same flags as Basic authentication.

• Command syntax: wget --user=myusername --password=mypassword https://example.com/digest-protected-file.zip

When executing this command, wget performs a multi-step handshake:

1. wget sends an initial request to the server without credentials.
2. The server rejects the request with a 401 Unauthorized status and includes a WWW-Authenticate: Digest ... header containing the nonce and realm.
3. wget intercepts this challenge, calculates the cryptographic hash using the provided password and nonce, and automatically sends a second request.
4. The second request includes an Authorization: Digest ... header containing the computed hash, verifying identity without exposing the password.

Securing Credentials with .wgetrc

Passing passwords directly into the command line creates a security risk, as your plaintext password will be visible to other users on the system via the process list (ps command) and will be saved in your shell’s command history file (e.g., .bash_history).

To prevent this, you can safely store your credentials in a private configuration file named .wgetrc located in your home directory.

1. Open or create the file: nano ~/.wgetrc
2. Add your authentication details:

http_user = myusername
http_password = mypassword

3. Restrict file permissions so only your user can read it: chmod 600 ~/.wgetrc

Once configured, you can run wget https://example.com/protected-file.zip without any authentication flags, and wget will automatically pull the necessary credentials from the file for both Basic and Digest challenges.