Cache Pollution and Memory Layout in C++

This article explores the concept of cache pollution, its detrimental impact on software performance, and the practical techniques C++ developers use to optimize memory layout. By understanding how hardware fetches data in fixed-size cache lines, developers can structure their types, split hot and cold data, and leverage data-oriented patterns like Struct of Arrays (SoA) to prevent memory bandwidth starvation and ensure that every byte brought into high-speed CPU caches serves an active computational purpose.

Understanding Cache Pollution

Modern processors rely on a hierarchy of small, extremely fast hardware caches (L1, L2, and L3) to bridge the speed gap between high-frequency CPU cores and comparatively slow main memory (RAM). When a CPU requests a specific memory address, it does not fetch only that requested byte. Instead, the hardware memory controller fetches a contiguous block of data called a cache line, which is typically 64 bytes wide on modern x86 and ARM architectures.

Cache pollution occurs when the cache is populated with data that is either never used or accessed infrequently, displacing data that the CPU needs immediately or repeatedly. Because cache capacity is severely limited, loading unneeded bytes pushes "hot" data out into slower memory tiers. This results in frequent cache misses, stall cycles while waiting for DRAM, and degraded throughput across the entire application.

How Memory Layout Causes Cache Pollution

In C++, the memory layout of classes, structs, and containers dictates which bytes reside next to one another in physical memory. Poorly planned layout directly causes cache pollution in two primary ways:

  1. Unused Sibling Data: If an algorithm repeatedly iterates over a specific field across a collection of objects, any adjacent member variables packed inside that same object are still pulled into the 64-byte cache line.
  2. Excessive Padding: Compilers insert padding bytes into structures to align member variables according to hardware constraints. If structs are arranged haphazardly, a significant portion of each cache line consists of dead padding.

Minimizing Unnecessary Loads in C++

To eliminate cache pollution, memory layouts must be organized around how the data is accessed, rather than strictly how entities are conceptually modeled.

1. Shifting from Array of Structures (AoS) to Struct of Arrays (SoA)

The traditional object-oriented paradigm naturally produces an Array of Structures (AoS):

struct Particle {
    float x, y, z;    // 12 bytes
    float vx, vy, vz; // 12 bytes
    int id;           // 4 bytes
    char name[32];    // 32 bytes
    // Total: ~60-64 bytes
};

std::vector<Particle> particles;

If a physics update loop only needs to update positions using velocities, iterating through particles will pull the id and name fields into the cache line alongside the coordinates. Over 50% of the cache bandwidth is wasted on data the loop never reads.

By restructuring the layout into a Struct of Arrays (SoA), related data is grouped together:

struct ParticleSystem {
    std::vector<float> x, y, z;
    std::vector<float> vx, vy, vz;
    std::vector<int> id;
    std::vector<std::string> name;
};

When updating physics, the processor reads only contiguous streams of floats. Every single byte in every loaded cache line is directly consumed by the arithmetic logic units, yielding peak cache efficiency and facilitating hardware vectorization (SIMD).

2. Hot/Cold Splitting

When an entity requires multiple fields that cannot easily be separated into an SoA model, developers can separate frequently accessed ("hot") members from rarely accessed ("cold") members.

// Poor layout: 64-byte cache line loads errors and metadata during every process() call
struct Connection {
    uint64_t socket_fd;     // Hot
    uint32_t bytes_sent;    // Hot
    uint32_t bytes_recv;    // Hot
    char error_log[256];    // Cold (only used on failure)
    char peer_cert[512];    // Cold
};

Separating these members into two structures ensures the hot structure fits snugly in a single cache line:

struct ConnectionColdData {
    char error_log[256];
    char peer_cert[512];
};

struct Connection {
    uint64_t socket_fd;
    uint32_t bytes_sent;
    uint32_t bytes_recv;
    std::unique_ptr<ConnectionColdData> cold_data; // Pointer to cold data
};

During regular execution loops, only the high-frequency metrics are read, preventing hundreds of bytes of diagnostic text from occupying critical L1/L2 cache slots.

3. Struct Member Ordering and Alignment

Compilers align data types to their natural boundaries. Placing smaller types between larger types introduces padding:

struct Inefficient {
    char a;    // 1 byte + 7 bytes padding
    double b;  // 8 bytes
    char c;    // 1 byte + 7 bytes padding
    double d;  // 8 bytes
}; // Total: 32 bytes (18 bytes of padding/dead space)

Reordering members from largest to smallest eliminates internal padding:

struct Efficient {
    double b;  // 8 bytes
    double d;  // 8 bytes
    char a;    // 1 byte
    char c;    // 1 byte
}; // Total: 18 bytes + 6 bytes tail padding = 24 bytes

This simple reorganization saves 8 bytes per instance, allowing 25% more elements to fit within the same cache footprint.

4. Preferring Contiguous Memory Over Node-Based Structures

Containers such as std::list, std::map, and pointer-heavy trees allocate nodes individually on the free store (heap). This produces pointer chasing, where nodes are scattered arbitrarily across memory.

Accessing a linked list node brings in a full 64-byte cache line just to read a single pointer and a payload value, evicting other data in the process. Replacing node-based containers with contiguous arrays (std::vector, std::deque, or flat maps) ensures that cache prefetchers can accurately predict upcoming addresses and saturate every cache line with meaningful application data.