Real-Time Audio Frequency Analysis in Three.js

This article provides a step-by-step guide on how to analyze audio frequencies in real-time using the AudioAnalyser class in Three.js. You will learn how to set up an audio listener, load an audio source, initialize the analyser, and retrieve frequency data to drive dynamic 3D visualisations.

1. Set Up the Audio Listener and Source

To analyze audio in Three.js, you first need an AudioListener and an Audio object. The listener acts as the “ears” of your scene and is typically added to the camera, while the audio object manages the playback.

import * as THREE from 'three';

// Create a camera and add the audio listener
const camera = new THREE.PerspectiveCamera(75, window.innerWidth / window.innerHeight, 0.1, 1000);
const listener = new THREE.AudioListener();
camera.add(listener);

// Create a global audio source
const sound = new THREE.Audio(listener);

2. Load the Audio File

Next, use the AudioLoader class to load an audio file. Once loaded, associate the audio buffer with your sound object and start playback.

const audioLoader = new THREE.AudioLoader();
audioLoader.load('path/to/your/audio.mp3', function(buffer) {
    sound.setBuffer(buffer);
    sound.setLoop(true);
    sound.setVolume(0.5);
    sound.play();
});

3. Initialize the AudioAnalyser

The AudioAnalyser class wraps the Web Audio API’s AnalyserNode. It extracts frequency data from the playing sound. Initialize it by passing your sound object and an fftSize (Fast Fourier Transform size). The fftSize determines the resolution of the frequency data and must be a non-zero power of two (e.g., 32, 64, 128, 256, 512).

const fftSize = 128; // Determines the frequency bin count (fftSize / 2)
const analyser = new THREE.AudioAnalyser(sound, fftSize);

4. Retrieve Frequency Data in the Animation Loop

To create real-time visual effects, query the analyser inside your requestAnimationFrame loop. You can retrieve raw frequency data as an array or get a single averaged value representing the overall volume.

• analyser.getFrequencyData(): Returns a Uint8Array containing frequency values ranging from 0 to 255.
• analyser.getAverageFrequency(): Returns a single average value of all frequencies, ideal for simple scale pulsations.

Here is how to apply this data to animate a 3D mesh:

const geometry = new THREE.BoxGeometry(1, 1, 1);
const material = new THREE.MeshBasicMaterial({ color: 0x00ff00 });
const cube = new THREE.Mesh(geometry, material);
scene.add(cube);

function animate() {
    requestAnimationFrame(animate);

    // Only analyze if the sound is playing
    if (sound.isPlaying) {
        // Option A: Get the average frequency
        const average = analyser.getAverageFrequency();
        
        // Scale the cube based on the audio average
        const scale = 1 + (average / 128); 
        cube.scale.set(scale, scale, scale);

        // Option B: Get detailed frequency data for custom vertex manipulation
        const dataArray = analyser.getFrequencyData();
        // dataArray[0] represents bass frequencies, dataArray[127] represents treble
    }

    renderer.render(scene, camera);
}

animate();