How FluidSynth Renders MIDI with SoundFonts

FluidSynth is a real-time, software-based synthesizer that converts Standard MIDI Files (SMF) or live MIDI streams into audible digital audio using sample-based SoundFont specifications. This article explains the underlying architecture and synthesis pipeline of FluidSynth, covering how it parses SoundFont files, processes MIDI commands, allocates synthesis voices, applies digital signal processing (DSP) parameters, and mixes the final audio output.

SoundFont Loading and Data Structuring

Before any audio is rendered, FluidSynth loads a SoundFont file (typically in SF2 or compressed SF3 format) into system memory. A SoundFont acts as a hierarchical container:

1. Samples: Raw, uncompressed PCM audio waveforms recorded from real instruments or synthesized sources.
2. Instruments: Collections of samples mapped across specific key ranges and velocity layers.
3. Presets: The user-facing patches (banks and program numbers) that group instruments together and define synthesis parameters known as "generators" (e.g., initial attenuation, filter cutoff, envelope timings) and "modulators" (e.g., routing velocity to filter frequency or LFO to pitch).

FluidSynth indexes these parameters according to the SoundFont 2.04 specification, preparing a lookup table that maps MIDI channels, bank numbers, and program changes directly to the correct sample data and articulation parameters.

MIDI Event Ingestion and Dispatching

When playing a MIDI file, FluidSynth reads timestamped MIDI events in sequence. The engine handles these messages through an internal event queue:

• Note-On: Triggers sound generation. FluidSynth looks at the current channel’s assigned bank and program, identifies the corresponding preset, and matches the note number and velocity to the correct sample zones.
• Control Change (CC): Adjusts real-time parameters such as volume (CC 7), pan (CC 10), expression (CC 11), sustain pedal (CC 64), and effect sends.
• Pitch Bend: Calculates microtonal pitch variations applied instantly to active notes.
• Program Change: Switches the active instrument preset for a specific MIDI channel.

Voice Allocation

For every Note-On event, FluidSynth creates one or more synthesis "voices." A single MIDI note can spawn multiple voices if the instrument zone is layered (for example, a stereo sample split across two independent channels, or multiple dynamic layers mapped to sound simultaneously).

FluidSynth manages these voices dynamically using an internal voice pool. If the number of concurrent voices exceeds the user-configured polyphony limit, the engine executes a voice-stealing algorithm, quietly terminating the quietest, oldest, or release-phase voices to free up DSP cycles.

The DSP Synthesis Pipeline

Once a voice is active, FluidSynth computes audio in discrete frames (buffers) through a multi-stage DSP process:

1. Wavetable Resampling: The raw sample rarely matches the exact pitch of the requested MIDI note. FluidSynth resamples the audio data on the fly to shift its pitch. To prevent aliasing and audio distortion, it uses selectable interpolation algorithms, ranging from basic linear interpolation to higher-order sinc interpolation.
2. Envelopes (DAHDSR): FluidSynth calculates two distinct envelopes per voice based on SoundFont parameters:
   • Volume Envelope: Shapes the amplitude over time (Delay, Attack, Hold, Decay, Sustain, Release).
   • Modulation Envelope: Dynamically modulates parameters such as pitch or filter cutoff over time.
3. Low-Pass Filtering: Each voice passes through an emulated resonant low-pass biquad filter. FluidSynth adjusts the cutoff frequency in real time based on envelope states, LFOs, and MIDI controller inputs.
4. Low-Frequency Oscillators (LFOs): The engine evaluates internal LFOs to generate effects like vibrato (pitch modulation) and tremolo (amplitude modulation).

Mixing and Effects Processing

After processing the individual synthesis voices, FluidSynth mixes the dry audio streams into internal audio buses. During this stage, channel panning and volume settings are applied.

The mixed signal is then routed into FluidSynth's integrated DSP effects units:

• Chorus: Modulates delayed copies of the signal to create a wider, richer stereo image.
• Reverb: Simulates acoustic space using an implementation of the Freeverb algorithm.

Output Generation

Finally, FluidSynth outputs the mixed 32-bit floating-point audio data. Depending on how the library is utilized, this audio is either sent directly to hardware audio drivers (such as ALSA, PulseAudio, JACK, CoreAudio, or WASAPI) for real-time playback, or written directly to disk as uncompressed PCM audio (such as a WAV or raw audio file) in fast offline rendering mode.