Why Unit Selection TTS Outperformed Diphone Synthesis

Unit selection Text-to-Speech (TTS) marked a monumental transition in speech synthesis by replacing rigid, unnatural-sounding diphone concatenation with dynamic, data-driven unit matching. While diphone synthesis relied on a minimal set of isolated acoustic fragments heavily modified by signal processing, unit selection mined large speech databases to find phonemes that naturally matched the desired context, pitch, and duration. This shift drastically eliminated digital processing artifacts, captured natural expressive prosody, and produced an unprecedented leap in synthesized voice realism during the late 1990s.

The Problem with Fixed-Inventory Diphone Synthesis

Before unit selection, the dominant concatenative approach was diphone synthesis. A diphone represents the transition from the middle of one phone to the middle of the next. To minimize computational demands and storage requirements, diphone synthesis operated on a fixed, minimal inventory. It contained roughly one or two recorded instances of each possible diphone pair in a given language (typically between 1,500 and 3,000 units).

Because these diphones were recorded with flat, neutral intonation, the TTS engine had to forcefully impose the target pitch (fundamental frequency) and duration onto the segments. This was accomplished using digital signal processing algorithms, most notably Pitch-Synchronous Overlap and Add (PSOLA). While computationally efficient, these algorithms severely degraded voice quality:

• Acoustic Artifacts: Continual stretching, compressing, and pitch-shifting introduced metallic, buzzy, and hollow artifacts into the speech.
• Loss of Natural Micro-Prosody: Human speech contains microscopic variations in spectral tilt, energy, and pitch that signal processing effectively erased, leaving voices sounding overtly robotic and lifeless.
• Phase Inconsistencies: Concatenating two arbitrary sound segments often resulted in spectral mismatches at the boundaries, leading to audible clicks and unnatural jumps.

The Breakthrough: Data-Driven Selection

Introduced primarily by researchers Andrew Hunt and Alan Black in 1996, unit selection turned the diphone philosophy upside down. Instead of collecting a minimal, static set of sound segments, unit selection leveraged large databases containing hours of naturally spoken, phonetically rich speech.

In this new paradigm, every phoneme, diphthong, and word existed in the database dozens or hundreds of times across diverse phonetic environments, pitch contours, and stress levels. Instead of modifying a sound to fit the target sentence, the engine searched the database to find an existing recording that already matched the target criteria.

Dual-Cost Optimization

The core mathematical breakthrough that powered unit selection was the formulation of a dynamic search problem solved via the Viterbi algorithm. The selection engine balanced two competing metrics:

1. Target Cost: How closely a candidate unit in the database matches the desired linguistic context, target pitch, duration, and position in the sentence.
2. Join Cost (Concatenation Cost): How smoothly two adjacent units will blend together at their boundary, measured through acoustic features like spectral distance and energy matching.

By calculating the path of lowest total cost across an entire utterance, the system selected sequences of units that sounded cohesive and natural together.

Minimizing Signal Processing

The defining advantage of unit selection was the near-total elimination of acoustic modification. Because the algorithm frequently found units that naturally possessed the intended duration and pitch, the synthesizer could splice segments directly together with little to no signal manipulation.

This preservation of pristine acoustic data ensured that the natural timbre of the speaker, natural formant transitions, and subtle acoustic cues remained intact. Longer units—such as entire words, syllables, or multi-phone sequences—could frequently be used without interruption, significantly reducing the total number of concatenation boundaries.

Summary of Impact

Unit selection transformed synthetic speech from an artificial, robotic utility into a lifelike representation of human voices. By trading minimal storage for large-scale data searching and algorithmic cost minimization, unit selection set the industry standard for naturalness and intelligibility for over two decades, laying the foundation for modern conversational interfaces and the deep learning speech technologies that followed.