Parse and Construct URLs with Python urllib.parse

Python's standard library provides the urllib.parse module to break uniform resource locators (URLs) into structured components and recombine them into valid strings. This article explains how to dissect URLs using functions like urlparse and urlsplit, handle query strings with parse_qs, and construct or modify URLs programmatically using urlunparse, urljoin, and urlencode.

Parsing URLs into Components

The primary function for breaking down a URL is urlparse(). It analyzes a URL string and separates it into six components based on standard syntax: scheme, netloc (network location/host), path, params, query, and fragment.

from urllib.parse import urlparse

url = "https://user:pass@example.com:8080/path/to/page;param?query=data#section"
parsed = urlparse(url)

print(parsed.scheme)    # 'https'
print(parsed.netloc)    # 'user:pass@example.com:8080'
print(parsed.path)      # '/path/to/page'
print(parsed.params)    # 'param'
print(parsed.query)     # 'query=data'
print(parsed.fragment)  # 'section'

Alternatively, urlsplit() operates similarly to urlparse(), but it does not separate standard path parameters into the params attribute. Instead, it leaves them within the path attribute, returning a 5-item named tuple. Modern applications and specifications generally prefer urlsplit() because modern web standards rarely treat the semicolon parameter segment separately from the path.

Both functions return named tuples, allowing component access via attribute names or numeric indexing.

Parsing Query Strings

The query component extracted by urlparse is a raw string. To convert this into usable Python data structures, urllib.parse provides parse_qs() and parse_qsl().

parse_qs(query_string): Parses parameters into a dictionary where each key maps to a list of values, preserving multiple identical keys.
parse_qsl(query_string): Parses parameters into a list of (key, value) tuples, preserving the original parameter order.

from urllib.parse import parse_qs, parse_qsl

query = "name=Alice&tag=python&tag=web"

print(parse_qs(query))
# Output: {'name': ['Alice'], 'tag': ['python', 'web']}

print(parse_qsl(query))
# Output: [('name', 'Alice'), ('tag', 'python'), ('tag', 'web')]

Constructing URLs

URLs can be assembled from individual components using urlunparse() or urlunsplit(). These functions require an iterable (such as a list, tuple, or the named tuple produced during parsing) containing the required number of components (6 for urlunparse, 5 for urlunsplit).

from urllib.parse import urlunparse, ParseResult

components = ParseResult(
    scheme="https",
    netloc="api.example.com",
    path="/v1/users",
    params="",
    query="active=true",
    fragment=""
)

constructed_url = urlunparse(components)
print(constructed_url)
# Output: https://api.example.com/v1/users?active=true

Creating Query Strings with urlencode

To build a query string from key-value pairs safely, use urlencode(). It automatically applies percent-encoding to special characters and spaces.

from urllib.parse import urlencode

params = {"search": "hello world", "page": 2, "filters": ["new", "featured"]}
query_string = urlencode(params, doseq=True)

print(query_string)
# Output: search=hello+world&page=2&filters=new&filters=featured

Setting doseq=True ensures that elements inside iterable values (like lists) are encoded as separate key-value pairs rather than stringified lists.

Joining Base and Relative URLs

When resolving links found within a document against the document's base address, urljoin() calculates the correct absolute URL according to RFC 3986 rules.

from urllib.parse import urljoin

base_url = "https://example.com/blog/articles/"

print(urljoin(base_url, "post-1.html"))
# Output: https://example.com/blog/articles/post-1.html

print(urljoin(base_url, "/images/pic.png"))
# Output: https://example.com/images/pic.png

print(urljoin(base_url, "../index.html"))
# Output: https://example.com/blog/index.html

Percent-Encoding and Decoding Strings

When inserting individual text fragments into paths or netlocs, quote() and quote_plus() replace unsafe characters with hex equivalents (e.g., %20). To reverse the process, use unquote() or unquote_plus().

from urllib.parse import quote, unquote

raw_string = "price: $10 & up"
encoded = quote(raw_string)
decoded = unquote(encoded)

print(encoded)  # price%3A%20%2410%20%26%20up
print(decoded)  # price: $10 & up

quote_plus() replaces spaces with + characters instead of %20, which is standard behavior when formatting query component values.