Parsing robots.txt with Python urllib.robotparser

The urllib.robotparser module in Python provides a standard framework for reading, parsing, and evaluating robots.txt files to ensure web crawlers respect website scraping policies. By using its core class, RobotFileParser, developers can automatically fetch a site's directives, interpret access restrictions for specific user agents, and determine whether a target URL is safe to crawl before sending request payloads.

How RobotFileParser Works

The parsing workflow within urllib.robotparser consists of three primary phases: fetching the file, parsing its contents into structured rules, and evaluating access permissions.

1. Fetching and Reading the File

To begin parsing, the RobotFileParser instance must be associated with the location of a target robots.txt file. This is handled using the set_url() method. When the read() method is executed, the module uses urllib.request to issue an HTTP GET request to retrieve the document:

import urllib.robotparser

rp = urllib.robotparser.RobotFileParser()
rp.set_url("https://www.example.com/robots.txt")
rp.read()

If you already have the contents of a robots.txt file as an iterable of strings (such as when reading from a local cache or a custom HTTP client), you can bypass network retrieval by using rp.parse(lines) instead of rp.read().

2. Parsing Directives

During the parsing phase, urllib.robotparser reads the file line-by-line, stripping whitespace and discarding comments (lines beginning with #). It processes standard directives by grouping them into user-agent-specific blocks:

• User-agent: Identifies which crawler the subsequent rules apply to. The parser checks for exact matches to your crawler's name or falls back to the wildcard (*) directive if no specific entry exists.
• Disallow: Marks URL paths that the specified user-agent must not access.
• Allow: Explicitly permits access to subpaths within an otherwise disallowed path.
• Crawl-delay: Indicates the number of seconds a crawler should wait between successive requests.
• Request-rate: Defines the request frequency limits (e.g., requests per number of seconds).

The parser stores these directives internally as rule objects, associating each set of Allow and Disallow path patterns with their corresponding user-agent definitions.

3. Checking URL Permissions

Once the directives are parsed into memory, the crawler queries whether a specific URL can be crawled using the can_fetch(useragent, url) method.

user_agent = "MyCustomBot"
target_url = "https://www.example.com/data/page.html"

if rp.can_fetch(user_agent, target_url):
    print("Access allowed: Proceeding with fetch.")
else:
    print("Access denied: Skipping URL.")

The can_fetch method operates using specific path comparison rules:

• It first looks for a User-agent block matching the query string. If no match is found, it falls back to the * block.
• It compares the target URL's path component against the Allow and Disallow entries sequentially.
• The most specific rule (usually the longest matching path) takes precedence. If an Allow and Disallow rule have identical lengths, the Allow rule typically overrides the Disallow rule.
• If a path does not match any specified rule, access is granted by default.

Handling Network Errors and Edge Cases

The parser applies default behaviors when encountering network failures:

• HTTP 401 or 403: If the server returns a forbidden or unauthorized status code for robots.txt, the parser assumes all access is restricted (can_fetch returns False).
• HTTP 404: If the robots.txt file does not exist, the parser assumes there are no access limitations (can_fetch returns True).
• HTTP 5xx: Server-side errors typically prevent the file from being read, leading the parser to treat access as fully restricted until the server recovers.