Measuring Speaker Similarity with Cosine Similarity

In zero-shot voice cloning Text-to-Speech (TTS), measuring how closely a synthesized voice matches an unseen target speaker is critical for assessing model performance. This article explains how speaker similarity is evaluated quantitatively through objective metrics, focusing on the extraction of high-dimensional speaker embeddings, the mathematical application of cosine similarity, the evaluation protocol, and the practical strengths and limitations of this approach.

The Role of Speaker Verification Models

Zero-shot voice cloning aims to replicate a target voice using only a short reference audio prompt that the model never encountered during training. To measure speaker similarity quantitatively without human listeners, researchers use pre-trained speaker verification (SV) networks.

Popular SV architectures include ECAPA-TDNN, ResNet-based models, and self-supervised models like WavLM or Wav2Vec 2.0. These models are trained on massive multi-speaker datasets (such as VoxCeleb) to project raw audio into a compact, fixed-dimensional latent space. The resulting vectors, commonly called speaker embeddings (or d-vectors/x-vectors), capture speaker-specific acoustic traits—such as vocal tract geometry and fundamental pitch range—while ignoring linguistic content, recording environments, and background noise.

The Cosine Similarity Metric

To quantify how identical the synthesized voice is to the target prompt, embeddings are extracted from both audio samples:

1. Reference Embedding (\(u\)): Extracted from the ground-truth reference audio provided to the cloning model.
2. Synthesized Embedding (\(v\)): Extracted from the generated audio produced by the TTS system.

The similarity between these two vectors is calculated using the cosine similarity formula:

\[\text{Cosine Similarity}(u, v) = \frac{u \cdot v}{\|u\|_2 \|v\|_2} = \frac{\sum_{i=1}^{n} u_i v_i}{\sqrt{\sum_{i=1}^{n} u_i^2} \sqrt{\sum_{i=1}^{n} v_i^2}}\]

Cosine similarity computes the cosine of the angle between the two vectors in multidimensional space.

• A value of 1.0 indicates identical orientation, representing maximum voice similarity.
• A value of 0.0 indicates orthogonality, meaning no correlation between the speaker profiles.
• Negative values indicate opposing vectors, though deep speaker embeddings generally fall within positive ranges for human speech.

Because cosine similarity evaluates the directional alignment rather than the magnitude of the vectors, it remains invariant to absolute volume and scaling differences between the audio tracks.

Quantitative Evaluation Protocol

A standard benchmarking workflow uses a held-out dataset of unseen speakers and follows a structured procedure:

1. Pair Generation: For each target speaker, one utterance is selected as the cloning prompt, and the model synthesizes a test sentence.
2. Score Extraction: The cosine similarity is calculated between the synthesized sample and the reference sample (or an alternate real recording of the same speaker to prevent trivial acoustic matching).
3. Thresholding and Baseline Comparison: Scores are benchmarked against two reference sets:
   • Positive Pairs: Different real recordings from the same human speaker to establish an empirical upper bound.
   • Negative Pairs: Real recordings from different speakers to establish a distinctiveness threshold.
4. Aggregation: Individual pair scores are averaged across the dataset to produce a single Mean Cosine Similarity (MCS) or Speaker Similarity Score (SIM). Often, models are also evaluated on Equal Error Rate (EER), assessing whether an automated verification system can accurately distinguish the clone from imposter voices based on a similarity threshold.

Strengths and Limitations

Cosine similarity provides a rapid, cost-effective, and fully automated evaluation method that correlates strongly with human Mean Opinion Score (MOS) evaluations for speaker identity.

However, the metric has notable constraints. It does not measure audio fidelity, naturalness, or speech intelligibility, which must be tested using complementary metrics such as Word Error Rate (WER) and Perceptual Evaluation of Speech Quality (PESQ). Additionally, if the synthesized speech contains acoustic artifacts or significant channel differences from the reference file, the embedding model may misinterpret these anomalies, artificially lowering or skewing the similarity score.