SIMD Vectorization in 8-Bit vs 10-Bit AV1 Arithmetic

Single Instruction, Multiple Data (SIMD) vectorization is the backbone of real-time AV1 encoding and decoding performance, allowing modern processors to manipulate multiple pixel values simultaneously. This article explores how SIMD vectorization accelerates AV1 arithmetic operations across 8-bit and 10-bit color depths. It covers register lane utilization, arithmetic expansion and overflow management, memory bandwidth constraints, and why 8-bit pipelines consistently achieve significantly higher mathematical throughput compared to their 10-bit counterparts.

Register Packing and Lane Utilization

The primary performance difference between 8-bit and 10-bit AV1 arithmetic in SIMD stems from data alignment within vector registers. Standard CPU architectures (such as x86 using AVX2/AVX-512 or ARM using NEON/SVE) feature vector registers sized at 128, 256, or 512 bits. These architectures provide native operations for 8-bit bytes, 16-bit half-words, 32-bit words, and 64-bit double-words, but they lack native support for 10-bit integers.

Because hardware registers do not support arbitrary 10-bit lanes, 10-bit AV1 pixel data must be stored within 16-bit containers. This introduces a structural throughput divide:

Even without considering mathematical complexity, 8-bit processing processes twice as many raw samples per cycle as 10-bit processing for purely element-wise operations like clipping, addition, and blending.

Arithmetic Promotion and Overflow Headroom

AV1 workloads—including the discrete cosine transform (DCT), asymmetric discrete sine transform (ADST), intra-prediction, and loop filters (such as CDEF and the loop restoration filter)—involve continuous accumulation and matrix multiplications. These arithmetic operations quickly exceed the original bit-depth of the input pixels, forcing algorithms to promote data types into larger intermediate containers to prevent numeric overflow.

In an 8-bit pipeline, intermediate products can frequently be accumulated in 16-bit registers. Instructions such as pmaddubsw (Multiply and Add Packed Unsigned and Signed Bytes) multiply pairs of 8-bit values and accumulate them into 16-bit signed integers in a single clock cycle. This allows 8-bit AV1 algorithms to maintain high vector density even during stages with intermediate expansion.

In a 10-bit pipeline, multiplying two values already requires intermediate headroom beyond 16 bits (up to 20 bits or more during accumulation). Consequently, the data must be unpacked and promoted from 16-bit containers into 32-bit integers. Promoting to 32-bit lanes cuts vector capacity in half again: a 256-bit register processes only 8 lanes at a time. The result is a substantial drop in arithmetic throughput, as operations require twice as many instructions and registers to compute the same number of pixel values.

Instruction Set Efficiency Across Transforms and Filters

The arithmetic efficiency gap is particularly visible in AV1's core computational modules:

1. Directional Intra Prediction

Intra-prediction involves linear interpolation between reference pixels. In 8-bit mode, fractional weighting operations can be handled using byte-level shifts and multiplies, often completing an entire 4x4 or 8x8 block prediction in a few instructions. In 10-bit mode, reference pixels must be loaded as 16-bit words, multiplied, shifted, and clamped, requiring roughly double the number of vector instructions to evaluate the same spatial area.

2. Constrained Directional Enhancement Filter (CDEF)

CDEF relies heavily on determining the primary direction of edges and applying non-linear filtering along that direction. The filter uses conditional thresholding and absolute difference operations. Because 8-bit values can use saturated arithmetic (paddusb, psubusb) natively across 32 or 64 lanes simultaneously, edge detection runs at near-peak hardware throughput. The 10-bit CDEF algorithm requires 16-bit arithmetic, increasing latency per vector execution block.

3. Loop Restoration Filter (Wiener and Self-Guided)

The Wiener filter requires separable symmetric convolutions across horizontal and vertical axes. Accumulating products over a 7-tap filter pushes mathematical values far beyond 8-bit boundaries. For 8-bit AV1, these taps fit neatly into 16-bit accumulation pathways. For 10-bit AV1, the intermediate sums often require 32-bit integer arithmetic to retain precision before final normalization, creating significant register pressure and spill-over into memory caches.

Memory Bandwidth and Cache Footprint

SIMD arithmetic units can only perform as fast as load and store instructions feed them. Because 10-bit data is padded into 16-bit words in memory to enable efficient SIMD vector alignment, a 10-bit frame uses 100% more memory bandwidth and CPU cache space than an unpadded 8-bit frame, despite carrying only 25% more dynamic range information.

This increased memory footprint reduces the efficiency of L1 and L2 data caches. An L1 data cache line (typically 64 bytes) stores 64 samples of 8-bit data, but only 32 samples of padded 10-bit data. The lower data density leads to higher cache miss rates during memory-bound operations like motion estimation, reference frame fetching, and motion compensation, causing the SIMD execution units to stall more frequently while waiting for memory operands.

Summary of Throughput Differences

SIMD vectorization accelerates both 8-bit and 10-bit AV1 pipelines, but architectural constraints favor 8-bit operations. By avoiding container waste and maintaining smaller intermediate footprints during arithmetic expansion, 8-bit AV1 pipelines achieve up to a 2x to 4x mathematical throughput advantage over 10-bit pipelines on equivalent SIMD hardware widths.