Statistical Tests for TTS Listening Tests

Subjective listening tests, such as Mean Opinion Score (MOS), MUSHRA, and AB preference tests, represent the benchmark for evaluating Text-to-Speech (TTS) audio quality and naturalness. However, reporting raw score increases without proper statistical testing undermines the validity of research findings. This article outlines the essential statistical significance tests required to rigorously substantiate perceived improvements in TTS systems, categorizing methods by experimental design, data distribution, and error correction.

Mean Opinion Score (MOS) Tests

In an absolute category rating (ACR) MOS test, listeners evaluate audio samples on an ordinal scale, typically ranging from 1 (Bad) to 5 (Excellent). Because these ratings represent discrete, ordinal data that rarely follow a normal distribution, standard parametric tests can produce misleading results.

• Mann-Whitney U Test (Wilcoxon Rank-Sum): Used when comparing two independent groups of listeners or unmatched stimuli. It assesses whether the distribution of ratings for one model stochastically dominates the other without assuming normality.
• Wilcoxon Signed-Rank Test: Preferred when the same pool of listeners evaluates identical text prompts across both the baseline and proposed systems (paired samples).
• Linear Mixed-Effects Models (LMM) and Cumulative Link Models (CLM): Advanced reporting increasingly demands mixed-effects models. These models treat the TTS system as a fixed effect while treating individual listeners and specific sentences as random effects, effectively controlling for listener bias and sentence difficulty variance.

MUSHRA Tests

MUlti Stimulus test with Hidden Reference and Anchor (MUSHRA) evaluations present listeners with multiple systems simultaneously on a continuous scale from 0 to 100. This within-subject design yields paired, continuous data.

• Repeated-Measures ANOVA: Applicable if the score differences pass normality checks (e.g., via the Shapiro-Wilk test). It evaluates whether significant differences exist across any of the evaluated systems.
• Friedman Test: The non-parametric alternative to repeated-measures ANOVA, utilized when MUSHRA scores violate normality assumptions.
• Post-Hoc Pairwise Comparisons: If the global test (ANOVA or Friedman) indicates significance, paired t-tests or Wilcoxon signed-rank tests must be conducted between individual system pairs to isolate where the improvement lies.

AB and ABX Preference Tests

Preference tests ask listeners to choose which of two samples sounds better, more natural, or more intelligible. The output produces categorical or binomial data.

• Two-Tailed Binomial Test: The standard requirement for forced-choice pairwise comparisons (A vs. B). It tests whether the proportion of preferences for the proposed system significantly deviates from a 50% chance baseline.
• Chi-Square Goodness-of-Fit Test: Utilized when an explicit "no preference" or "tie" category is permitted, testing whether the distribution across the three outcomes deviates from expected proportions.

Multiple Comparisons Corrections

When comparing three or more TTS architectures against one another or against a shared baseline, conducting multiple pairwise tests increases the probability of Type I errors (false positives). Applying a correction method is mandatory:

• Bonferroni Correction: A conservative adjustment that divides the significance threshold (typically \(\alpha = 0.05\)) by the total number of pairwise comparisons.
• Holm-Bonferroni Method: A step-down approach that maintains the family-wise error rate control of Bonferroni while offering greater statistical power.
• Benjamini-Hochberg Procedure: Controls the False Discovery Rate (FDR) instead of the family-wise error rate, suitable for large-scale comparative benchmarks involving many synthetic voices.

Effect Sizes and Confidence Intervals

Statistical significance denotes only that an observed difference is unlikely due to random chance; it does not measure practical relevance. Comprehensive TTS reporting requires:

• Confidence Intervals (CI): Typically reported at the 95% level alongside the mean or median to display the margin of error around the perceived score.
• Standardized Effect Sizes: Metrics such as Cohen’s \(d\) (for parametric comparisons) or Cliff’s Delta and Rank-Biserial Correlation (for ordinal comparisons) quantify the magnitude of the audio quality gain, providing critical context for whether an improvement is audible and meaningful to real-world users.