What Is BorgBackup and How Does It Deduplicate Data?

BorgBackup, commonly known as Borg, is an open-source, deduplicating backup tool designed for Linux and Unix-like operating systems that provides secure, space-efficient storage for critical data. This article explores what BorgBackup is, highlights its primary features, and explains the content-defined chunking process it uses to eliminate redundant data across backup archives.

What is BorgBackup?

BorgBackup is a command-line backup program built with an emphasis on speed, security, and storage efficiency. Unlike traditional backup utilities that create full or differential image archives, Borg stores data in a dedicated repository where individual backups—termed "archives"—function as complete snapshots while only consuming storage for newly introduced changes.

Key features of BorgBackup include:

• Authenticated Encryption: Client-side encryption secures all data using AES-256 or ChaCha20-Poly1305 before it leaves the host machine.
• Data Compression: Files can be compressed using algorithms like LZ4, Zstandard, Zlib, or LZMA prior to storage.
• Remote Storage: Backups can be pushed to remote servers via SSH without needing dedicated daemon services beyond standard SSH access.
• FUSE Mounting: Archives can be mounted as local filesystems, allowing users to inspect and restore individual files seamlessly using standard file managers or command-line utilities.

How BorgBackup Deduplicates Data

Borg achieves exceptional storage savings through chunk-based data deduplication. Rather than comparing entire files or using rigid block boundaries, Borg evaluates incoming data streams dynamically to detect repeated content across different files, directories, and past backups.

1. Content-Defined Chunking

Borg splits files into variable-length pieces called chunks using a rolling hash algorithm (specifically, a cyclic polynomial algorithm known as Buzhash). As a file is read, the rolling hash slides byte-by-byte over the data stream. When the hash value matches a specific mathematical pattern, a chunk boundary is declared.

Because chunk boundaries are determined by the content itself rather than fixed offsets:

• Adding or deleting bytes at the beginning or middle of a file only changes the chunks immediately surrounding the modification.
• The remaining chunks retain identical boundaries, ensuring the rest of the file continues to match previously stored data.

2. Cryptographic Hashing and Chunk Identification

Once a chunk is created, Borg computes an authentication hash or HMAC (such as HMAC-SHA256 or BLAKE2b) for that specific block of data. This hash functions as the chunk's unique fingerprint ID.

3. Repository-Wide Index Lookup

Borg maintains a repository-wide index that tracks every unique chunk ID that has already been stored. Before saving a chunk:

• Borg checks the chunk ID against this local cache and the repository index.
• If the chunk ID already exists, the data is identified as duplicate. Borg skips storing the data, increments a reference counter, and merely adds a pointer to the existing chunk in the archive's metadata.
• If the chunk ID is not found, the chunk is treated as new data. It is compressed, encrypted, and written to the repository.

4. Cross-Archive Deduplication

Deduplication in Borg is global across the entire repository. This means:

• Identical files stored in different locations on the system are saved only once.
• Unaltered files between consecutive backup runs consume zero additional storage.
• Renaming or moving large directories does not cause data to be re-uploaded or duplicated.
• Different machines sharing a repository (or backups of multiple similar virtual environments) can deduplicate against each other if configured properly.

By combining content-defined chunking with cryptographic indexing, BorgBackup provides Linux systems with efficient daily backups, drastically lowering bandwidth requirements and disk footprint.