How 7-Zip Processes Sparse Files Without Allocating Space

Sparse files contain large blocks of empty data (zeros) represented by file system metadata rather than physical drive sectors. 7-Zip detects, compresses, and extracts these files efficiently by interfacing directly with underlying operating system APIs. During both archive creation and extraction, 7-Zip avoids allocating unneeded physical storage by handling non-zero data segments explicitly and instructing the file system to mark empty blocks as virtual holes.

Detecting and Compressing Sparse Files

When 7-Zip scans an input file system to create an archive, it checks file attributes to identify sparse files:

  1. Querying Allocated Ranges: On Windows (NTFS), 7-Zip queries the file system using the FSCTL_QUERY_ALLOCATED_RANGES control code via DeviceIoControl. This API returns an array detailing only the byte ranges that contain actual data, completely skipping the unallocated "holes."
  2. Reading Data Streams: Instead of reading millions of zeroed bytes sequentially, 7-Zip can skip directly between allocated offsets. If continuous zero streams are read, the LZMA and LZMA2 compression algorithms encode these repetitive sequences into negligible amounts of metadata.
  3. Archive Metadata Storage: 7-Zip stores sparse attributes and offset maps inside the .7z container, ensuring the structure can be accurately reproduced when extracted on a compatible file system.

Extracting Sparse Files Without Consuming Disk Space

The primary challenge occurs during extraction, where naive file writers write actual zeros to the disk, causing the file system to allocate physical clusters. 7-Zip avoids this through the following process on NTFS:

  1. Enabling the Sparse Flag: After creating the destination file handle, 7-Zip invokes DeviceIoControl with the FSCTL_SET_SPARSE flag. This signals to NTFS that the file should support unallocated regions.
  2. Writing Only Non-Zero Data: 7-Zip streams the extracted payload and positions the write pointer using SetFilePointerEx to target only the offsets containing actual data. It then writes those non-zero segments using standard WriteFile calls.
  3. Punching Zero Holes: If a block contains zeros, 7-Zip can either skip the range or explicitly call FSCTL_SET_ZERO_DATA. This tells NTFS to decommit or omit physical cluster allocation for that specific range while updating the metadata to reflect that the range contains zeros.
  4. Setting the Logical File Size: Finally, 7-Zip moves the file pointer to the end of the intended file size and calls SetEndOfFile. The operating system reports the large logical size in directory listings, while the physical size on disk reflects only the non-zero clusters written.

On Linux and POSIX environments (via POSIX-compatible builds like p7zip), 7-Zip employs standard sparse-file mechanisms such as lseek to jump past holes, or fallocate with the FALLOC_FL_PUNCH_HOLE flag, allowing the kernel to preserve sparse regions seamlessly.