rav1e Custom Assembly for x86 AVX2 and ARM

This article examines how the rav1e AV1 video encoder implements and integrates custom assembly for x86 AVX2 and ARM platforms. It covers the build toolchain integration, foreign function interface (FFI) bindings, architecture-specific assembly designs, and the runtime CPU detection mechanism used to safely dispatch hand-optimized vector operations.

Build Toolchain and Assembly Compilation

rav1e is primarily written in Rust, but it relies on low-level assembly routines to accelerate compute-heavy video encoding tasks such as discrete cosine transforms, motion estimation, loop filters, and pixel prediction. The compilation of these routines is managed during the build stage using a custom build.rs script.

• x86_64 (AVX2): Assembly routines are written in Netwide Assembler (NASM) syntax. The build script leverages the nasm-rs crate to detect a local installation of NASM or Yasm, compiling .asm source files directly into native object files that are linked into the final Rust binary.
• ARM / AArch64: ARM routines are written in standard GNU Assembler (GAS) syntax within .S files. The build script uses the cc crate to invoke the platform's native C compiler (such as GCC or Clang) to assemble these files with appropriate architecture flags (e.g., -march=armv8-a).

Architecture-Specific Vector Implementation

x86 AVX2 Routines

The x86 assembly routines in rav1e often derive from or share design patterns with the dav1d AV1 decoder. They utilize the widely adopted x86inc.asm macro abstraction layer, which provides:

• ABI Normalization: Automatically handles calling convention differences between System V (Linux/macOS) and Microsoft x64 (Windows), managing stack alignment and register preservation.
• YMM Register Allocation: Enables seamless access to 256-bit AVX2 vector registers (ymm0–ymm15), implementing SIMD parallelism on 8-bit and 10-bit pixel blocks.
• Fused Instructions: Employs AVX2 features including fused multiply-add (FMA3), broad bitwise manipulation, and 256-bit wide integer permutations to accelerate matrix transformations and pixel averaging.

ARM (NEON) Routines

ARM optimization focuses primarily on the 64-bit AArch64 platform using NEON vector extensions:

• Direct Vector Register Usage: Routines explicitly manage NEON's thirty-two 128-bit vector registers (v0–v31), operating simultaneously on 8-bit, 16-bit, or 32-bit lanes.
• Instruction Efficiency: Utilizes NEON instructions such as vector widening/narrowing arithmetic (saddl, uaddl), pairwise additions, and cross-lane permutations tailored for pixel-level prediction and filtering.
• Calling Convention Compliance: Adheres to the standard AAPCS64 ABI, directly mapping parameters passed from the Rust runtime into AArch64 general-purpose (x0–x7) and vector (v0–v7) registers.

Rust FFI and Safe Abstraction Layers

To integrate these external routines, rav1e defines C-compatible foreign function signatures within unsafe Rust modules:

extern "C" {
    fn rav1e_subpixel_filter_avx2(
        dst: *mut u8,
        dst_stride: isize,
        src: *const u8,
        src_stride: isize,
        width: i32,
        height: i32,
    );
}

Direct calls to these symbols are inherently unsafe. To prevent undefined behavior, rav1e encapsulates each assembly invocation within safe Rust wrappers. These wrappers validate slice boundaries, enforce correct memory alignments (often 16 or 32 bytes), and verify that input parameters conform strictly to block-size constraints required by the vector code.

Dynamic Runtime CPU Dispatch

rav1e supports diverse hardware targets by decoupling the assembly implementations from the baseline compiler target. It uses runtime CPU feature detection rather than relying exclusively on static compile-time flags.

1. Hardware Detection: At runtime or encoder initialization, rav1e queries the host CPU capabilities. On x86, it evaluates CPUID flags for instruction sets like AVX2, FMA3, and SSSE3. On ARM, it reads OS-provided capabilities or hardware auxiliary vectors (e.g., via getauxval on Linux) to check for NEON or advanced crypto/vector extensions.
2. Function Pointer Resolution: rav1e structures its core operations around function tables or trait implementations. During initialization, the encoder checks the detected feature flags and populates these tables with pointers to the fastest available implementation.
3. Fallback Strategy: If the required SIMD extensions (such as AVX2 on an older x86 machine) are absent, the dispatch mechanism falls back to auto-vectorized, pure Rust implementations. This design guarantees broad binary portability while maximizing performance on supported microarchitectures.