Minimum Torrent Overhead for Small Files

Generating a .torrent file for a directory of small files introduces metadata overhead determined by the BitTorrent specification's Bencode format. The absolute minimum overhead for a multi-file torrent consists of a static base metadata size of approximately 150 to 250 bytes, a piece hash requirement of at least 20 bytes, and an incremental overhead of roughly 25 to 50 bytes per file entry in the directory.

Core Components of Torrent Overhead

A .torrent file is a Bencoded dictionary. When sharing a directory containing multiple files (multi-file mode), the file size overhead comes from three main areas:

  1. Base Metadata: Contains static dictionary keys such as announce (tracker URL), creation date, created by, and the top-level info container dictionary. This accounts for roughly 100 to 200 bytes depending on the tracker URL length.

  2. Piece Hashes (pieces field): Data is split into fixed-size chunks (typically 16 KiB to 4 MiB). Each piece requires a 20-byte SHA-1 hash in the metadata. Even if the entire directory of small files totals less than a single piece, there is a mandatory minimum overhead of 20 bytes for the single SHA-1 hash.

  3. Multi-File Path Entries (files list): In multi-file mode, the info dictionary contains a files list. Each individual file adds a sub-dictionary containing its length and relative file path:

    d6:lengthi<file_size>e4:pathl<name_length>:<filename>ee

Mathematical Breakdown of Minimum Overhead

To calculate the theoretical minimum overhead for a directory containing \(N\) small files:

  • Static Base Structure: \(\approx 150\text{ bytes}\) (minimal tracker URL, piece length, and directory name).
  • Piece Hash Storage: \(\ge 20\text{ bytes}\) (assuming total content fits into a single piece).
  • Per-File Cost: \(22\text{ bytes} + \text{path string length}\). For a single-character filename (e.g., a.txt), this is roughly 27 bytes per file.

The formula for minimum overhead \(S\) in bytes is:

\[S \approx 170 + \sum_{i=1}^{N} (22 + \text{len}(\text{filename}_i))\]

Practical Overhead Examples

  • 1 single-byte file in a directory: Total .torrent file size is approximately 200 to 300 bytes.
  • 100 small files (average 10-character filenames): The per-file overhead adds approximately 3,200 bytes, resulting in a .torrent file size of approximately 3.4 KB to 3.6 KB.
  • 10,000 small files: The per-file metadata will scale linearly, producing a .torrent file of approximately 320 KB to 350 KB, regardless of how small the actual file contents are.

Summary

The minimum base overhead for creating a multi-file torrent starts at around 170 to 200 bytes. However, when dealing with large numbers of small files, the per-file dictionary entries dominate the overhead, adding roughly 30 to 40 bytes per file.