How H.265 and AV1 Handle UDP Packet Loss

Modern video codecs like H.265 (HEVC) and AV1 are engineered to maintain high-quality streaming over User Datagram Protocol (UDP), a transport protocol favored for real-time communication because of its low latency, despite its lack of guaranteed packet delivery. When transmitting video over UDP, lost packets can cause severe visual artifacts, frame freezing, and cascading decoding errors. To prevent visual degradation, advanced codecs integrate built-in error resilience tools, adaptive reference frame management, spatial isolation, and tight feedback loops with real-time transport protocols.

Temporal Scalability and Hierarchical Prediction

Modern codecs structure video frames into hierarchical layers. Instead of every P-frame or B-frame relying sequentially on the immediately preceding frame, codecs use temporal Scalable Video Coding (SVC) and hierarchical prediction structures. If an enhancement layer packet is lost in transit, the decoder drops that specific sub-layer without corrupting the base layer. The video stream continues playing at a temporarily reduced frame rate instead of freezing or producing macroblocking artifacts.

Long-Term Reference Frames and Feedback Mechanisms

When operating alongside protocols like WebRTC and RTP/RTCP, codecs adapt using signaling mechanisms: * Picture Loss Indication (PLI) and Full Intra Request (FIR): When packet loss causes unrecoverable frame corruption, the receiver sends feedback requesting an immediate recovery frame. * Long-Term Reference (LTR) Frames: Rather than sending a bandwidth-heavy keyframe (I-frame) upon packet loss, codecs can mark specific, verified frames as long-term references. When a packet is lost, the encoder instructs the decoder to reference an older, intact LTR frame, immediately halting error propagation without a major bitrate spike. * AV1 S-Frames (Switch Frames): AV1 introduces switch frames, allowing decoders to switch resolutions or recover from loss by referencing higher-layer frames without decoding the entire sequence from a keyframe.

Intra-Refresh vs. Massive Keyframes

Traditional recovery involves transmitting an I-frame, which requires significantly more data than a standard inter-frame and can exacerbate UDP network congestion. Modern implementations use Periodic Intra Refresh (PIR). Instead of refreshing the entire image at once, a narrow vertical or horizontal column of intra-coded blocks sweeps across the screen over several frames. This spreads the recovery bitrate evenly across the stream, healing loss-induced artifacts gradually without congesting the connection.

Spatial Partitioning and Independent Decoding Units

To isolate the impact of packet loss within a single frame, both H.265 and AV1 split frames into independently decodable spatial regions: * H.265 Slices and Wavefront Parallel Processing (WPP): Frames are divided into slices that can be decoded independently. If a UDP packet containing one slice is dropped, adjacent slices remain fully decodable. * AV1 Tiles: AV1 relies on a grid-based tile architecture. Tiles operate with independent context and prediction dependencies. Packet loss affecting one tile is confined to its specific spatial boundary, preventing corruption from spreading across the entire frame.

Decoder-Side Error Concealment

When packet loss occurs and recovery packets cannot arrive in time, modern decoders execute error concealment algorithms: * Spatial Concealment: The decoder interpolates missing pixels using boundary pixels from adjacent, successfully decoded blocks within the same frame. * Temporal Concealment: The decoder uses motion vectors from previous undamaged frames to estimate the missing block’s content, effectively carrying forward motion smoothly until a refreshed reference frame arrives.