How 7-Zip Computes CRC-32 and CRC-64 Checksums

This article explains how the 7-Zip file archiver calculates CRC-32 and CRC-64 checksums to detect data corruption and ensure file integrity. It covers the underlying mathematical principles of cyclic redundancy checks, software optimization techniques like multi-table slicing, modern hardware acceleration, and the internal workflow 7-Zip uses when testing archives or calculating hashes on demand.

Understanding Cyclic Redundancy Checks (CRC)

A Cyclic Redundancy Check (CRC) is an error-detecting code based on polynomial division. 7-Zip treats the incoming stream of file bytes as the coefficients of a massive binary polynomial. This binary polynomial is divided by a fixed standard generator polynomial using modulo-2 arithmetic (where addition and subtraction are equivalent to the bitwise XOR operation).

The mathematical remainder left after processing the entire byte stream forms the checksum:

  • CRC-32: Uses a 33-bit generator polynomial to yield a 32-bit (4-byte) remainder. 7-Zip uses the widely adopted IEEE 802.3 standard polynomial (\(0x04C11DB7\) normal, or \(0xEDB88320\) in bit-reversed representation).
  • CRC-64: Uses a 65-bit generator polynomial to yield a 64-bit (8-byte) remainder. 7-Zip utilizes the ECMA-182 standard polynomial (\(0x42F0E1EBA9EA3693\) normal, or \(0xC96C5795D7870F42\) in bit-reversed representation).

The initial state begins with all bits inverted (0xFFFFFFFF for CRC-32 and 0xFFFFFFFFFFFFFFFF for CRC-64) to detect prepended zero bytes, and the final remainder is bit-inverted (XORed with all ones) before being displayed to the user.

Software Optimization: Precomputed Lookup Tables

Calculating polynomial division bit-by-bit is computationally expensive. To maximize software performance across all architectures, 7-Zip uses multi-byte table-driven algorithms commonly known as "Slicing-by-8" or "Slicing-by-16":

  1. Precomputed Tables: During compilation or initial runtime, 7-Zip precalculates tables that map combinations of input bytes to their resulting remainder values.
  2. Multi-Byte Processing: Instead of processing one byte at a time through a 256-entry table, the slicing algorithm reads 8 or 16 bytes of data simultaneously from the input buffer.
  3. Pipelining: The algorithm splits the multi-byte block across several parallel lookup tables, XORs the results together, and updates the running CRC value. This technique keeps CPU execution pipelines filled and drastically reduces memory fetch bottlenecks.

Hardware Acceleration

On modern processors, 7-Zip bypasses standard table lookups in favor of dedicated hardware instructions:

  • x86/x64 Architectures: 7-Zip can leverage the PCLMULQDQ (Carry-Less Multiplication) instruction set alongside AVX/SSE vector instructions. Carry-less multiplication computes the polynomial division across large 128-bit or 256-bit vectors in parallel, achieving processing speeds of several gigabytes per second.
  • ARM Architectures: 7-Zip utilizes ARMv8 CRC32 hardware extensions (CRC32B, CRC32W, CRC32X), which perform cycle-accurate checksum calculations natively in silicon.

The Integrity Verification Workflow

When you test an archive or use the "CRC SHA" context menu in 7-Zip, the application follows this process:

  1. Buffer Streaming: The target file is read into memory in sequential blocks (typically 64 KB to a few megabytes) to minimize disk I/O overhead.
  2. Continuous Accumulation: Each block is fed into the active CRC engine (hardware-accelerated if available, table-sliced if not), which updates the global state variable.
  3. Finalization: Once the end of the file or stream is reached, the internal state is inverted to generate the final hexadecimal string.
  4. Header Comparison: For archive verification (such as .7z or .zip), 7-Zip compares the freshly computed CRC against the CRC value stored in the archive’s metadata header when the file was originally compressed. If the computed checksum and the stored checksum match, the file is confirmed to be uncorrupted; if they differ, 7-Zip reports a data error.