PESQ and POLQA in Text-to-Speech Evaluation

Perceptual Evaluation of Speech Quality (PESQ) and Perceptual Objective Listening Quality Analysis (POLQA) are standardized objective metrics originally engineered to predict human Mean Opinion Scores (MOS) in telecommunications. While researchers frequently attempt to use them to benchmark Text-to-Speech (TTS) systems, their utility in this domain is highly constrained. PESQ and POLQA apply effectively only to isolated subcomponents of a synthesis pipeline—such as vocoder reconstruction against known ground-truth audio—but fundamentally fail as end-to-end TTS metrics due to their reliance on time-aligned reference signals, inability to score prosody or expressiveness, and insensitivity to generative synthesis artifacts.

The Design Purpose of PESQ and POLQA

PESQ (ITU-T P.862) and its successor POLQA (ITU-T P.863) are full-reference perceptual quality algorithms. They compare a degraded audio signal against an original, pristine reference signal to quantify quality degradation introduced by telecommunication networks, speech codecs (such as AMR or Opus), packet loss, and jitter. Both models simulate human psychoacoustics—mapping signals into loudness and pitch representations to calculate perceptual distance scores that correlate with subjective listening tests.

Where PESQ and POLQA Apply to TTS

Because these algorithms require a reference signal, their application to TTS is limited strictly to scenarios featuring a direct acoustic ground truth:

• Vocoder Quality Assessment: Neural vocoders (such as HiFi-GAN or WaveGlow) synthesize time-domain waveforms directly from intermediate representations like mel-spectrograms. When evaluating a vocoder in an "analysis-synthesis" loop, the input mel-spectrogram is extracted from an actual human recording. The vocoder's reconstructed audio can then be directly compared to the original human recording. In this specific setup, PESQ and POLQA measure the spectral fidelity and distortion introduced purely by the vocoder.
• Acoustic Codec and Compression Benchmarking: When deploying TTS engines into telecommunications applications (such as Interactive Voice Response systems), PESQ and POLQA can evaluate how downstream audio compression and transmission protocols affect the intelligibility and degradation of the synthesized audio stream.

Where PESQ and POLQA Fail to Apply

Despite their success in network telephony, PESQ and POLQA break down when applied to general or end-to-end TTS evaluation:

• The Lack of a Ground Truth: Generating speech from raw text creates an acoustic realization from scratch. There is no natural, uncompressed reference recording for an arbitrary text prompt. Without a direct acoustic counterpart, full-reference metrics cannot function.
• Intolerance to Temporal and Prosodic Variation: Even if a human reference speaker records the same text prompt, normal variations in pacing, phoneme duration, pauses, and intonation cause massive temporal misalignments. Although POLQA features improved time-alignment algorithms over PESQ, neither model can distinguish between acceptable expressive variation and destructive signal timing errors. An entirely natural-sounding synthetic sentence with a slightly slower cadence will receive an artificially low quality score.
• Inability to Evaluate Naturalness and Expressiveness: PESQ and POLQA evaluate signal degradation, not human-likeness. A robotic, flat, or emotionally disjointed voice can exhibit pristine spectral clarity and receive a near-perfect POLQA score, despite sounding unnatural to human listeners.
• Irrelevant Distortion Models: Telephony metrics look for specific artifacts: clipping, background noise, bandwidth restrictions, and packet drops. Modern neural TTS models produce entirely different failure modes, such as phoneme hallucinations, unnatural pitch transitions, metallic buzz, phase smearing, or unnatural breathing patterns. Full-reference telecom models are not weighted to penalize these generative errors accurately.

Preferred Alternatives for TTS Quality Evaluation

Due to the architectural limitations of PESQ and POLQA, TTS assessment relies on other methods:

• Subjective Testing: Human listening evaluations—specifically MOS (Mean Opinion Score), MUSHRA (MUlti-Stimulus test with Hidden Reference and Anchor), and side-by-side preference tests—remain the benchmark for naturalness, intelligibility, and speaker similarity.
• No-Reference (Non-Intrusive) Objective Models: Deep learning-based metrics such as NISQA, UTMOS, and MOSNet assess speech quality and naturalness directly from the synthesized audio without requiring a reference signal.
• ASR-Driven Metrics: Automated Speech Recognition systems transcribing generated speech to calculate Word Error Rate (WER) or Character Error Rate (CER) serve as objective, reliable proxies for intelligibility.