Why BitTorrent Uses Concatenated SHA-1 Pieces Hashes

In BitTorrent metainfo (.torrent) files, the pieces key maps to a single, contiguous byte string of concatenated 20-byte SHA-1 hashes rather than a Bencoded list of individual strings. This article explains the technical reasons behind this design decision, focusing on metadata size reduction, parsing performance, and memory efficiency in BitTorrent clients.

Bencode Encoding Overhead

BitTorrent uses Bencode to serialize data structures. In Bencode, a list of strings requires each item to have a length prefix and delimiter. Storing each 20-byte SHA-1 hash as an element in a list would look like:

l20:<20-byte hash>20:<20-byte hash>...e

Each hash in a list format incurs an additional 3 bytes of overhead (20:). For large torrents with tens of thousands of pieces, this overhead quickly inflates the .torrent file size by 15%.

By representing the hashes as a single contiguous binary string, Bencode only needs a single length prefix:

<total_length>:<concatenated hashes>

This ensures the metadata remains compact, reducing file transfer sizes across trackers, peers, and indexing sites.

Fast and Direct Memory Indexing

A contiguous string aligns directly with native memory buffers. When a client reads the pieces string, it loads the entire sequence into a single flat array of bytes.

Accessing the verification hash for any piece index N requires a simple constant-time pointer calculation:

\[\text{Offset} = N \times 20\]

The client slices bytes from Offset to Offset + 20 without traversing list pointers, iterating over node objects, or handling separate memory allocations for each piece.

Parser Simplicity and Performance

Decoding thousands of individual elements in a Bencoded list requires the parser to perform thousands of string allocations, pointer assignments, and memory checks. In low-power devices, routers, or high-throughput seedboxes, this adds measurable CPU and garbage collection overhead.

Treating all hashes as a single byte array allows Bencode parsers to allocate memory once, copy the payload in a single operation, and avoid heap fragmentation.

Strict Validation

Storing hashes contiguously guarantees format integrity through length verification. Because SHA-1 produces a 160-bit (20-byte) output, a client can instantly validate the entire pieces field by checking if the total byte length is divisible by 20:

\[\text{Length} \pmod{20} = 0\]

If the byte length fails this check, the client immediately rejects the .torrent file as corrupted or malformed.