Evaluating TTS Voice Similarity with Metric Learning

Metric learning provides the algorithmic foundation for objectively assessing how closely a cloned Text-to-Speech (TTS) voice matches an authentic reference voice. By projecting complex audio signals into a structured, low-dimensional embedding space, metric learning enables automated systems to calculate a mathematical distance between two voices. This article explores how metric learning works in this context, the architectures and loss functions that power it, and its critical role in benchmarking modern neural voice synthesis.

The Challenge of Objective Voice Evaluation

Evaluating voice similarity in zero-shot or few-shot TTS is difficult because speech contains multiple overlapping layers of information, including linguistic content, background noise, emotion, and speaker identity. Traditional audio metrics like Mean Squared Error (MSE) or Signal-to-Noise Ratio (SNR) measure raw waveform differences, making them useless for comparing a reference utterance to a cloned voice reading entirely different text. While Mean Opinion Score (MOS) listening tests offer reliable human judgment, they are slow, expensive, and difficult to scale.

Metric learning solves this by teaching deep neural networks to isolate speaker identity and ignore text, pitch variance, and channel acoustics.

How Metric Learning Functions in Voice Comparison

In metric learning, a neural network acts as an encoder that maps raw audio or spectrograms into a fixed-length vector called an embedding (such as a d-vector or x-vector). The core goal is to structure the embedding space so that:

• Embeddings from the same speaker have minimal distance (high similarity).
• Embeddings from different speakers have maximal distance (low similarity).

To evaluate TTS output, the trained encoder extracts an embedding from the original reference audio and another from the synthesized TTS audio. The system then computes a geometric metric—most commonly cosine similarity:

\[\text{Cosine Similarity} = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\| \|\mathbf{v}\|}\]

Where \(\mathbf{u}\) represents the reference embedding and \(\mathbf{v}\) represents the cloned audio embedding. A score closer to 1 indicates that the synthesized voice shares the acoustic characteristics of the target speaker.

Key Architectures and Loss Functions

Metric learning systems for voice evaluation rely on deep speaker verification architectures trained on massive voice datasets (such as VoxCeleb). Prominent backbones include:

• ECAPA-TDNN: An enhanced Time-Delay Neural Network utilizing squeeze-and-excitation blocks to capture both local and global temporal acoustic features.
• ResNet-based Encoders: Standard deep residual networks adapted to process 2D log-mel spectrograms.

These networks are optimized using specialized margin-based loss functions rather than standard classification loss:

• Triplet Loss: Enforces a margin between an anchor sample, a positive sample (same speaker), and a negative sample (different speaker).
• Additive Angular Margin Loss (ArcFace / CosFace): Introduces an angular margin penalty directly to the target classes on a hypersphere, producing highly discriminative speaker embeddings that generalize well to unseen speakers.

Practical Applications in Voice Synthesis

Within modern TTS pipelines, metric learning fulfills several vital functions:

1. Automated Quality Assurance: Developers can automatically test synthesized voices against reference samples across thousands of test sentences to identify voice drift or identity collapse.
2. Loss Formulation During Training: Many state-of-the-art cloning models integrate a pretrained speaker encoder directly into the training loop, using the metric distance as a perceptual loss to guide the generator toward a closer vocal match.
3. Equal Error Rate (EER) Benchmarking: By treating cloned samples as genuine attempts to spoof a speaker verification model, researchers determine how effectively a cloned voice can fool biometric security, establishing a standardized measure of vocal authenticity.