FFmpeg Aresample Audio Rate Conversion

The aresample filter in FFmpeg is the primary tool for changing the sample rate, channel layout, and sample format of audio streams. This article explains how this filter handles sample rate conversions under the hood, exploring its default resampling engines, mathematical interpolation methods, and how it manages audio-to-video synchronization through dynamic sample rate adjustments.

Resampling Engines: Swr vs. SoX

Under the hood, the aresample filter acts as a wrapper for two primary resampling libraries: libswresample (the default FFmpeg resampler) and libsoxr (the SoX Resampler library).

• swr (Software Resampler): This is the default engine. It is highly optimized for speed and uses polyphase filterbank interpolation. It is suitable for most real-time streaming and general transcoding tasks.
• soxr (SoX Resampler): This engine must be explicitly enabled. It uses high-quality, band-limited interpolation algorithms. While it requires more CPU power, it offers superior audio quality with minimal aliasing, making it the preferred choice for high-fidelity audio archiving.

You can specify the engine using the resampler option: -af aresample=resampler=soxr

The Mathematical Process: Anti-Aliasing and Interpolation

When converting from a source sample rate (e.g., 44.1 kHz) to a target sample rate (e.g., 48 kHz), aresample performs three primary mathematical operations:

1. Up-sampling / Down-sampling: The filter inserts zero-valued samples between existing samples (upsampling) or discards samples (downsampling) to approximate the target rate.
2. Low-Pass Filtering: To prevent aliasing (where high-frequency sounds fold down into lower, audible frequencies), a Kaiser-windowed sinc low-pass filter is applied. By default, the cutoff frequency is set slightly below the Nyquist frequency of the lower of the two sample rates.
3. Polyphase Filtering: Because exact integer ratios between rates (like 44100 and 48000) would require massive filter sizes, aresample uses a polyphase filterbank. It interpolates between the closest calculated samples using a configurable number of phases (controlled by the phase_shift parameter) to determine the exact amplitude of the new sample.

Managing Audio Synchronization (Async and Compensation)

In addition to static rate conversion, aresample is crucial for maintaining audio-to-video synchronization. When audio packets arrive with timestamps that do not match the expected timeline, aresample can dynamically stretch or squeeze the audio.

• async: This parameter enables synchronization stretching. If timestamps drift, the filter dynamically alters the output sample rate slightly to catch up or slow down, avoiding hard audio cuts.
• min_comp and max_comp: These settings define the threshold for sample rate compensation. If the time drift is small, aresample stretches the audio smoothly. If the drift exceeds the specified limit, it performs a hard drop or inserts silence (controlled by min_hard_comp).

Common Configuration Parameters

To fine-tune how aresample handles your audio, you can adjust several key parameters in your filter string:

• osr (Output Sample Rate): Sets the target sample rate (e.g., osr=48000).
• filter_size (Length): Controls the length of the resampling filter. Higher values increase frequency precision but require more processing time.
• cutoff: Sets the filter cutoff frequency ratio. The default is typically 0.91 for swr and 0.95 for soxr. Lowering this value reduces aliasing but cuts off high frequencies earlier.
• dither_method: When lowering bit depth alongside sample rates, this adds noise shape dither (e.g., triangular or rectangular) to prevent quantization distortion.