How Multi-File Torrents Manage File Hierarchies

Multi-file torrents manage complex directory structures by encoding folder hierarchies and file paths directly into the metadata of the .torrent file or magnet link data. By utilizing the Bencode data serialization format, the BitTorrent protocol stores directories as structured lists of strings, maps these paths to specific file lengths, and treats the entire folder structure as a single, continuous stream of data for transfer.

The Bencoded info Dictionary

At the core of every torrent file is the info dictionary, serialized using Bencoding. When a torrent contains multiple files, the info dictionary alters its schema compared to a single-file torrent:

• name: In a multi-file torrent, the top-level name key represents the name of the root directory rather than an individual file.
• files: Instead of a single length key, a multi-file torrent contains a files list. Each entry in this list is a dictionary containing metadata for an individual file.

Representing Subdirectories with the path List

BitTorrent achieves platform independence by avoiding operating-system-specific path separators like / on Unix/macOS or \ on Windows. Instead, inside each dictionary in the files list, the torrent defines:

1. length: An integer indicating the exact size of the file in bytes.
2. path: A list of strings where each element represents a step in the directory tree, terminating with the file name.

For example, a file located at RootFolder/Music/Rock/track1.mp3 with a size of 5 MB is encoded in the metadata as:

{
    "length": 5242880,
    "path": ["Music", "Rock", "track1.mp3"]
}

The client reads this list sequentially, treating every item before the final element as a nested directory, and the final element as the actual file name.

Continuous Byte Stream Mapping

While the metadata maintains a clear visual and logical directory tree, the BitTorrent transfer engine abstracts the entire file hierarchy into a single, continuous stream of bytes.

The protocol concatenates all files in the order they appear in the files list. It then divides this continuous byte stream into uniform chunks known as “pieces” (typically between 256 KB and 16 MB). Because pieces are split based purely on byte offsets:

• A single piece may contain the end of one file and the beginning of another.
• Pieces frequently span across directory boundaries.
• The torrent protocol does not transfer “folders”; it transfers indexed pieces and uses the metadata map to write the downloaded byte ranges to their corresponding local files.

Client-Side Construction and Path Sanitization

When a download begins, the BitTorrent client parses the files list and constructs the local folder hierarchy using the host operating system’s native file system APIs.

During this process, the client performs path sanitization: * Illegal Characters: Strips or replaces characters not supported by the local OS (such as :, *, ?, ", <, >, | on Windows). * Path Traversal Prevention: Strips directory traversal tokens like ../ or absolute path markers to prevent files from being written outside the intended download directory. * Path Length Limitations: Truncates or renames paths that exceed operating system path length limits.

Once the directory skeleton is created on the storage drive, the client writes the incoming byte blocks into their correct file offsets within the generated folder tree.