Measuring CPU Core-to-Core Latency Across Chiplets and CCXs

This article explains the methodology behind measuring CPU core-to-core latency across different Core Complexes (CCXs) and chiplets during hardware benchmarking. It covers the fundamental software techniques used to force inter-core communication, the role of thread affinity, how hardware timers track nanosecond delays through cache coherency protocols, and how these measurements reveal the underlying physical architecture of modern processors.

Thread Affinity and Isolation

To measure the latency between two specific CPU cores, a benchmarking program must first ensure that the operating system does not move the testing tasks around. Benchmarks achieve this using thread affinity (or thread pinning).

Using operating system APIs, the software creates two separate threads and locks them to two specific logical or physical cores—for example, Core 0 and Core 1. This guarantees that the communication being measured is strictly between those two exact hardware locations.

The "Ping-Pong" Atomic Test

CPU cores do not send direct messages to one another in standard workloads; instead, they communicate by reading and writing to shared memory and caches. To measure latency, benchmarks use a "ping-pong" test that forces a piece of data (a cache line) to bounce back and forth between the two pinned cores.

The process works like this:

  1. Initialization: A shared variable is created in the system memory, which is loaded into the CPU's cache.
  2. The Ping: Core A uses an atomic instruction, such as Compare-And-Swap (CAS), to change the value of the shared variable from 0 to 1.
  3. Cache Invalidation: Because Core A modified the data, the CPU’s cache coherency protocol invalidates that data in Core B's cache.
  4. The Pong: Core B is running a tight loop, constantly checking the variable. Once it detects the value has changed to 1, it fetches the updated cache line, and immediately uses an atomic instruction to change the value back to 0.
  5. Repetition: Core A detects the change back to 0 and repeats the process.

Because the cores must wait for the hardware to validate and transfer the cache line before they can execute the next change, the speed of this loop is entirely bottlenecked by the physical latency between the two cores.

High-Precision Timing

A single transfer between cores happens in just a few nanoseconds, which is too fast to measure accurately on its own due to software overhead. To get an accurate reading, the benchmark bounces the data back and forth thousands or millions of times.

Before the loop starts, the software records the current time using the CPU's internal hardware timer, typically accessed via the RDTSC (Read Time-Stamp Counter) assembly instruction. Once the million bounces are complete, it reads the timer again.

The total elapsed time is then divided by the number of transfers to calculate the average latency in nanoseconds. Depending on the benchmark, this is reported either as a "round-trip" latency (Core A to Core B and back) or divided by two to represent "one-way" latency.

Crossing CCXs and Chiplets

The benchmark does not inherently know the physical layout of the CPU. It simply runs the ping-pong test across every possible pair of cores (Core 0 to Core 1, Core 0 to Core 2, etc.) and plots the results in a grid. The physical architecture reveals itself through the data:

  • Intra-CCX (Same Core Complex): If Core A and Core B are on the same CCX, they share a local L3 cache. The cache line only needs to travel across the local L3 ring or bus. The benchmark will record very low latency (typically 10 to 30 nanoseconds).
  • Inter-CCX (Different Core Complexes, Same Chiplet): In older architectures where a single silicon die was split into two CCXs, the data had to leave one L3 cache, traverse an internal interconnect, and enter the other L3 cache. This results in a noticeable jump in latency.
  • Inter-Chiplet / CCD (Different Silicon Dies): If Core A is on Chiplet 1 and Core B is on Chiplet 2, the cache line must physically leave the silicon die, travel across the package substrate (such as AMD's Infinity Fabric) to the central I/O die, and then route to the second chiplet. The benchmark captures this long physical journey as a massive spike in latency (often 70 to 100+ nanoseconds).

By automating this test across all core combinations, hardware reviewers generate heatmaps that clearly show the boundaries of CCXs, chiplets, and Simultaneous Multithreading (SMT) sibling threads based purely on the time it takes to bounce a single cache line back and forth.