How Torrent Parsers Handle Dictionaries and Bytes

A .torrent file uses a compact serialization format called Bencode to store metadata about shared files and trackers. This article explains the technical mechanics of how a Bencode parser processes raw byte strings and nested dictionary structures. It covers the parsing algorithms for length-prefixed binary data, the use of recursion and state machines for nested structures, and the critical need to preserve raw byte boundaries for cryptographic hash verification.

Understanding the Bencode Format

Before examining dictionaries and byte strings, it is necessary to understand the four primary data types defined by the BitTorrent specification:

  1. Integers: Wrapped in i and e (e.g., i42e).
  2. Byte Strings: Formatted as <length>:<data> (e.g., 4:spam).
  3. Lists: Wrapped in l and e (e.g., l4:spami42ee).
  4. Dictionaries: Wrapped in d and e containing alternating keys and values.

Because Bencode uses ASCII delimiters alongside raw binary payloads, a parser cannot simply decode the entire file as a standard UTF-8 string. It must process the input as an arbitrary array of bytes.

Parsing Byte Strings

Byte strings in a .torrent file are length-prefixed. The parser identifies a byte string when it encounters an ASCII digit (0 through 9).

The Parsing Algorithm:

  1. Read the Length: The parser reads ASCII digits sequentially until it hits the delimiter character : (ASCII 0x3A).
  2. Convert to Integer: The parsed ASCII characters are converted to an integer, representing the byte count \(N\).
  3. Extract Payload: The parser advances its cursor past the colon and reads exactly \(N\) consecutive bytes directly from the byte stream into a byte buffer.
Raw Data:   5 : h e l l o
Cursor:    [0]  ^ (Length = 5)
            [1]    ^ (Delimiter)
            [2..6] ^ (Read 5 raw bytes: "hello")

Handling Binary vs. Text Data

Not all byte strings represent human-readable text. For example, the pieces field inside the info dictionary contains concatenated 20-byte SHA-1 hashes of each file chunk. A correct parser must store string values as raw byte arrays rather than automatically decoding them into text, preventing encoding errors or corrupted hashes when encountering arbitrary binary data.

Parsing Nested Dictionaries

Dictionaries start with the byte d (ASCII 0x64) and terminate with the byte e (ASCII 0x65). They consist of alternating key-value pairs where: - Keys must be Bencoded byte strings. - Keys must appear in lexicographical (binary) order. - Values can be any valid Bencode data type, including another dictionary or list.

Handling Nesting with Recursive Descent

When a parser encounters a d marker, it enters a dictionary-parsing state:

  1. The parser initializes an empty map or associative array.
  2. It loops until it encounters the terminator byte e.
  3. In each iteration:
    • It parses the next element, ensuring it is a byte string (the dictionary key).
    • It parses the following element as the value. If the value starts with d, the parser makes a recursive call to the dictionary parser.
    • The key-value pair is stored in the current map.
  4. When the matching e is reached, the parser returns the populated dictionary and advances past the terminator.
Raw Stream:  d 4:info d 4:name 4:file e e
Structure:   {
               "info": {
                 "name": "file"
               }
             }

Iterative Parsing Using an Explicit Stack

In environments where deep recursion might cause stack overflow issues, parsers use an explicit stack to maintain state: - When a d is found, a new dictionary container is pushed onto the stack. - The parser tracks whether it expects a key or a value for the container at the top of the stack. - When an e is encountered, the container at the top of the stack is popped and attached to the parent structure beneath it.

Preserving Raw Offsets for the Info Hash

A unique requirement when parsing .torrent dictionaries is calculating the info_hash. The BitTorrent protocol identifies torrents by the SHA-1 hash of the bencoded info dictionary.

Because dictionary key ordering, whitespace, and integer representations can vary if re-encoded naively by a library, standard parsers must record the exact start and end byte offsets of the info dictionary within the raw file. The parser slices these raw bytes directly from the source buffer and passes them through a SHA-1 hashing function, guaranteeing an accurate info_hash matching the torrent network.