AV1 Decoder Vulnerabilities Uncovered by Fuzzing

Early implementations of the AV1 video codec, notably reference libraries like AOMedia’s libaom and the VideoLAN-led dav1d project, were subjected to extensive automated testing through fuzzing frameworks such as Google's OSS-Fuzz, AFL, and LibFuzzer. This article outlines the primary security vulnerabilities exposed by these fuzzing tools, detailing how issues such as out-of-bounds memory operations, integer overflows, unhandled edge cases, and memory exhaustion bugs were identified and remediated before AV1 achieved ubiquitous deployment across modern web browsers and operating systems.

Out-of-Bounds Reads and Writes

The most prevalent class of vulnerabilities identified by fuzzers involved memory safety violations, specifically out-of-bounds (OOB) heap reads and writes. Video decoders must parse dense, variable-length bitstreams and map them into fixed memory structures.

Fuzzing engines systematically generated malformed bitstreams with mutated Open Bitstream Unit (OBU) headers, corrupted tile lists, and invalid partition trees. These inputs caused decoders to calculate incorrect array indices during motion vector prediction, inter/intra-frame interpolation, and coefficient decoding. In several instances within libaom, an unexpected block size or transform type caused pointers to write past the bounds of allocated frame buffers, creating potential avenues for arbitrary code execution or process crashes.

Integer Overflows and Calculation Wraparounds

The AV1 specification introduced advanced features such as dynamic frame scaling, super-resolution, and complex loop restoration filters (such as Wiener filters and self-guided restoration). These algorithms rely on fixed-point arithmetic and coordinate scaling calculations.

Fuzzers repeatedly exposed integer overflow and underflow conditions in early decoder codebases. Malformed headers specifying extreme or non-standard frame dimensions, cropping parameters, or scaling factors caused arithmetic operations to wrap around. These calculations frequently determined the size of subsequent memory allocations. When an integer overflow caused a memory allocator to receive a truncated size value, it resulted in undersized buffers that were immediately overrun by the decoder, yielding critical heap-based buffer overflows.

Unhandled Parsing States and Null Pointer Dereferences

AV1 bitstreams are divided into discrete units (OBUs) that define sequence configurations, metadata, frame headers, and tile groups. Fuzzing tools tested unexpected sequencing, omission, and duplication of these units.

Early decoder versions frequently assumed a strict, compliant order of syntax elements. By supplying mutated files where crucial configuration headers (such as sequence headers or color configuration metadata) were missing or prematurely terminated, fuzzers triggered null pointer dereferences and assertion failures. While rarely exploitable for code execution, these parsing bugs consistently produced reproducible remote Denial-of-Service (DoS) crashes in applications embedding the decoders, including Chromium and Mozilla Firefox.

Loop Filtering and Boundary Violations

AV1 employs three distinct in-loop filtering stages: the deblocking filter, the Constrained Directional Enhancement Filter (CDEF), and the loop restoration filter. These filters operate across block and tile boundaries and require synchronization with neighboring reconstruction buffers.

Fuzzing uncovered multiple edge cases where boundary checks failed, particularly around non-standard frame edges, 128x128 superblocks, or when tiles were decoded in parallel across multiple worker threads. When mutated bitstreams signaled filter strengths or directional parameters on incomplete frame borders, decoders occasionally sampled uninitialized memory or attempted to read pixels beyond the reference picture boundaries.

Resource Exhaustion and Algorithmic Complexity

Beyond traditional memory corruption, fuzzing frameworks uncovered algorithmic complexity vulnerabilities and excessive resource allocation states.

Attackers could construct miniature bitstreams—often only a few kilobytes in size—that instructed decoders to allocate gigantic multi-plane reference frame buffers or instantiate an unsustainable number of spatial and temporal operating points. Fuzzers caught instances where decoders attempted memory allocations before validating that the bitstream actually contained the corresponding frame payload, making host systems susceptible to out-of-memory (OOM) denial-of-service conditions.

Continuous integration with automated fuzzing platforms ultimately identified hundreds of low-level memory defects and logic errors in early AV1 decoders. This continuous testing hardened libraries like libaom and dav1d, establishing rigorous input validation boundaries and bounds checking prior to their widespread adoption in modern consumer hardware and web platforms.