How Linux Interacts with ZFS Adaptive Replacement Cache

This article explores how the Linux operating system interacts with the ZFS Adaptive Replacement Cache (ARC). It details how OpenZFS manages its memory footprint independently of the standard Linux page cache, the mechanisms used by the Linux kernel to signal memory pressure to ZFS, and how system administrators can tune this interaction to prevent system instability and maximize storage throughput.

Bypassing the Linux Page Cache

In a standard Linux setup, the kernel uses available system RAM for the Linux page cache, which buffers reads and writes for filesystems like ext4 and XFS. ZFS on Linux (OpenZFS) behaves fundamentally differently. It bypasses the native Linux page cache for file data, routing read and write operations through its own caching mechanism: the Adaptive Replacement Cache (ARC).

The ARC is designed to balance both recency (Most Recently Used / MRU) and frequency (Most Frequently Used / MFU) of data access. Because the ARC exists as a separate layer originally designed for Solaris, ZFS uses the Solaris Porting Layer (SPL) within Linux to interface with the host kernel’s memory subsystems.

Memory Allocation via the Linux Kernel

The ARC allocates memory directly from the Linux kernel using standard internal allocators:

• Slab/SLUB Allocators: ZFS requests kernel memory slabs to manage metadata and buffer headers.
• vmalloc and Virtual Memory Subsystems: Larger, contiguous memory allocations required for data blocks are handled via the kernel's virtual memory subsystem.

Because the ARC consumes kernel memory rather than standard user-space buffers or page cache pages, tools like free or top historically report ARC usage as active kernel slab memory (SUnreclaim or SReclaimable) rather than "buff/cache." In modern OpenZFS releases, memory is marked to appear more accurately in reporting tools, but it remains strictly managed by ZFS rather than Linux's virtual filesystem (VFS) page cache layer.

Memory Pressure and the Linux Kernel Shrinker

The most critical interaction between Linux and the ARC occurs when the system faces memory exhaustion. Because the Linux kernel cannot directly free ZFS ARC buffers on its own, it relies on a cooperative mechanism called a shrinker callback.

1. Memory Contention Detection: When user-space applications request memory and free RAM drops below specific watermarks, the Linux virtual memory manager (kswapd) initiates memory reclamation.
2. Invoking the ZFS Shrinker: The Linux kernel invokes registered shrinker routines, including the one exposed by the OpenZFS module.
3. Releasing ARC Buffers: Upon receiving the shrinker event, the ARC dynamically prunes data—evicting first from the MRU or MFU lists based on internal eviction algorithms—and returns the freed memory pages directly to the Linux kernel.
4. Latency Challenges: While this process is automatic, asynchronous shrinker calls can occasionally lag behind sudden, massive memory allocation spikes by user applications, which can lead to Out-Of-Memory (OOM) killer invocations if the ARC cannot shrink quickly enough.

Tuning ARC Limits in Linux

By default, OpenZFS on Linux configures the maximum ARC size (zfs_arc_max) to half of the total physical RAM on modern systems (or up to 75% on older releases). Administrators can control how aggressively ZFS consumes Linux memory by modifying OpenZFS module parameters at runtime or within /etc/modprobe.d/zfs.conf.

• zfs_arc_max: Sets the upper ceiling for ARC memory consumption, preventing ZFS from starving memory-intensive applications such as databases or virtual machines.
• zfs_arc_min: Establishes a floor below which the Linux shrinker cannot force the ARC to drop, preserving critical storage metadata performance.
• arc_shrink_shift: Dictates the aggressiveness of the ARC eviction rate during memory reclamation cycles.

Adjusting these settings allows the Linux kernel and the ZFS ARC to coexist predictably, balancing raw filesystem caching performance against the working memory needs of the rest of the operating system.