How Torrent Caches Preserve Metadata Without Indexers

This article explains how online torrent caching services collect, store, and preserve torrent metadata independently of centralized indexing websites. By utilizing decentralized networking protocols—primarily the Distributed Hash Table (DHT) and metadata exchange extensions—these services autonomously discover and archive .torrent files using only unique cryptographic info hashes, ensuring that torrent data remains accessible even when original indexers or trackers go offline.

The Foundation: Cryptographic Info Hashes

Every BitTorrent transfer relies on an info hash, a unique cryptographic identifier (typically SHA-1 in BitTorrent v1 or SHA-256 in v2) calculated from the info dictionary of a .torrent file.

Because the info hash uniquely identifies the content and file structure rather than the location of the file, it serves as a universal key. Caching services do not need website databases or descriptions; they only require the info hash to identify, locate, and reconstruct the exact metadata file.

DHT Scraping and Network Crawling

Online caches primarily bypass centralized indexers by participating directly in the BitTorrent Mainline Distributed Hash Table (MLDHT) based on the Kademlia algorithm.

1. Node Participation: Caching services deploy specialized DHT crawler nodes into the network.
2. Ping and Route Discovery: These nodes continuously traverse the routing tables of other DHT nodes, monitoring get_peers and announce_peer queries.
3. Hash Harvesting: Whenever a user announces or searches for a torrent via DHT, the crawler records the active info hash.

Through continuous network crawling, caches discover virtually all publicly shared torrents in real time without visiting a single indexing website.

Fetching Metadata via BEP 9 and BEP 10

Knowing an info hash only confirms a torrent’s existence; it does not contain the actual metadata (file list, piece hashes, and directory structure). To preserve the full .torrent file, caching services fetch the metadata directly from active peers using the BitTorrent Extension Protocol.

• Extension Negotiation (BEP 10): The cache node connects to peers discovered via the DHT and sends an extended handshake to negotiate supported features.
• Metadata Transfer (BEP 9 / ut_metadata): If the peer supports the metadata extension, the cache requests the metadata in 16 KiB chunks.
• Validation and Assembly: Once all chunks are received, the cache verifies the data against the original info hash. If the hash matches, the raw metadata is assembled into a standard, valid .torrent file.

Independent Storage and Distribution

Once generated, the metadata is archived in high-speed, persistent databases indexed purely by info hash.

• Magnet Link Resolution: When a user or client requests a magnet link, caching services can instantly serve the complete .torrent file using the hash, skipping the time-consuming process of querying the DHT swarm for metadata.
• Permanent Archival: Even if the original upload website shuts down or the swarm temporarily loses seeders, the caching service retains the file structure and piece hashes, preserving the torrent’s blueprint indefinitely.