How L1 and L2 CPU Caches Use Spatial and Temporal Locality

This article explores how modern processors overcome the significant speed gap between fast CPU cores and slower main memory by utilizing multi-level L1 and L2 caches. By relying on the fundamental computing principles of temporal and spatial locality, these small, high-speed memory banks accurately predict which data and instructions the processor will need next. Understanding this relationship reveals how electronic systems minimize latency, prevent data bottlenecks, and drastically accelerate overall execution speeds.

The Memory Bottleneck

In modern electronics, Central Processing Units (CPUs) process instructions at billions of cycles per second. However, the system's main memory, or Dynamic Random-Access Memory (DRAM), operates at a fraction of that speed. If the CPU had to fetch every instruction and piece of data directly from the main memory, it would spend most of its time sitting idle, waiting for data to arrive.

To solve this bottleneck, hardware engineers place ultra-fast Static Random-Access Memory (SRAM) directly onto the processor chip. This is the CPU cache. Because SRAM is physically larger and far more expensive to manufacture than DRAM, the cache must be kept small. Since it can only hold a tiny fraction of the data stored in main memory, the cache must be exceptionally smart about what data it chooses to keep. It achieves this efficiency through the principles of locality.

What is Temporal Locality?

Temporal locality is the principle that if a specific memory location is accessed at a given point in time, it is highly likely to be accessed again in the near future.

In software execution, this happens constantly. Consider a program running a loop:

  • The same set of instructions is executed repeatedly until the loop finishes.
  • Specific variables, such as a loop counter or a running total, are read and updated during every single iteration.

When the CPU accesses this data for the first time, it fetches it from the slow main memory and stores a copy in the cache. Because of temporal locality, the cache retains this data. For every subsequent run of the loop, the CPU grabs the instructions and variables directly from the fast cache, completely bypassing the slow main memory.

What is Spatial Locality?

Spatial locality is the principle that if a specific memory location is accessed, nearby memory locations are highly likely to be accessed soon after.

Computer programs are rarely scattered randomly across memory. Instead, they are organized sequentially:

  • Instructions are generally executed one right after the other in a linear order.
  • Data structures, such as arrays, store related items in contiguous, side-by-side memory blocks.

Caches exploit spatial locality by never fetching just a single byte of data. Instead, when the CPU requests a specific piece of data, the cache fetches a larger, contiguous chunk of memory surrounding that data, known as a "cache line" (typically 64 bytes). When the CPU inevitably asks for the next sequential instruction or the next item in an array, that data is already waiting inside the cache.

How Multi-Level L1 and L2 Caches Work Together

Because of the physical laws of electronics, a larger memory bank takes longer to search and requires signals to travel further distances, which introduces delay. If engineers built a single, massive cache, it would be too slow to keep up with the CPU core. To balance speed and capacity, processors use a multi-level hierarchy, primarily consisting of Level 1 (L1) and Level 2 (L2) caches.

Level 1 (L1) Cache

The L1 cache is the smallest (often between 32KB and 64KB per core) and the fastest memory in the computer, located closest to the CPU's execution unit. It usually operates at the exact same speed as the processor itself.

Because it is so small, L1 relies heavily on temporal locality. It holds the absolute most critical data and instructions that the CPU is actively using right now. To maximize speed, L1 is often split into two parts: L1i (for instructions) and L1d (for data), allowing the processor to fetch both simultaneously.

Level 2 (L2) Cache

When the CPU needs data that is not in the L1 cache (an "L1 cache miss"), it checks the L2 cache. The L2 cache is significantly larger (ranging from 256KB to several megabytes) and sits slightly further away from the core, making it marginally slower than L1, yet still vastly faster than main memory.

Because of its larger size, the L2 cache acts as a safety net that heavily exploits spatial locality and broader temporal locality. It holds the larger blocks of prefetched cache lines, entire arrays, and recently used data that got pushed out of the tiny L1 cache to make room for newer tasks.

The Execution Speedup

By combining the L1 and L2 architectures with the principles of locality, the CPU achieves a "cache hit rate" of over 90% in most computing tasks.

  1. The CPU requests data.
  2. Temporal locality ensures that if the data was just used, it is instantly available in the L1 cache, taking only a few nanoseconds.
  3. Spatial locality ensures that as the program moves forward, the next required data was already pulled into the L1 or L2 cache as part of a larger cache line.
  4. Only when a program jumps to a completely new, unpredicted task does a "cache miss" occur, forcing the system to endure the long wait of fetching data from main memory.

Without spatial and temporal locality, the small L1 and L2 caches would be guessing blindly at what data to store, rendering them useless. Because software naturally exhibits these two behaviors, multi-level caches can accurately predict the CPU's needs, keeping the execution pipeline full and allowing modern electronics to run at peak performance.