TTS Text Normalization Accuracy Benchmarks

Accurate verbalization of non-standard words—such as complex numbers, calendar dates, and currency values—is a critical component of Text-to-Speech (TTS) pipelines known as Text Normalization (TN). This article outlines the primary benchmarks, standardized datasets, and evaluation methodologies used by machine learning researchers to measure how reliably TTS front-ends and end-to-end models convert written semiotic tokens into spoken equivalents.

The Google Text Normalization Dataset

The most widely used benchmark for evaluating the verbalization of numbers, dates, and currencies is the Google Text Normalization Dataset, introduced by Richard Sproat and Navdeep Jaitly. Derived from Wikipedia, this corpus covers English and Russian, categorizing non-standard tokens into explicit semiotic classes.

Key classes within the benchmark include:

• DATE: Covers complex date patterns, abbreviations, and relative temporal expressions (e.g., "Jan. 21, 2024" to "January twenty-first twenty twenty-four").
• MONEY: Validates currency symbols, fractional values, and international codes (e.g., "$3.50" to "three dollars and fifty cents", "£10M" to "ten million pounds").
• CARDINAL and ORDINAL: Tests integers, negative numbers, large scientific counts, and positional sequences.
• DECIMAL, FRACTION, and MEASURE: Evaluates fractional units, percentages, and dimension markers across varying contexts.

The dataset provides both the raw input text and the target verbalized transcript, allowing token-level and sentence-level accuracy comparisons.

NVIDIA NeMo Text Normalization (TN) Suite

NVIDIA’s NeMo framework provides comprehensive evaluation pipelines and rule-augmented benchmarks for both Text Normalization and Inverse Text Normalization (ITN). NeMo evaluates models against standard benchmark splits as well as specialized multilingual test sets across English, German, Spanish, Mandarin, and other languages.

The NeMo benchmark tests both weighted finite-state transducers (WFSTs) and neural models (such as Transformer-based sequence-to-sequence systems), offering standardized test scripts to calculate exact-match accuracy specifically isolated to numerical, currency, and temporal spans.

The NSW (Non-Standard Word) Benchmark

Historically established by the Johns Hopkins 2001 workshop on Non-Standard Word Normalization, the NSW corpus remains a foundational benchmark for classifying and verbalizing tokens that cannot be found in a standard phonetic dictionary. It specifically penalizes errors in ambiguous formats, such as determining whether a four-digit string represents a year ("1998" -> "nineteen ninety-eight") or a quantity ("1,998" -> "one thousand nine hundred ninety-eight").

End-to-End ASR-Based Verbalization Tests

Modern end-to-end neural TTS systems often bypass explicit text normalization modules, converting raw text directly into audio. Because traditional string-matching metrics cannot evaluate these systems directly, researchers employ Automatic Speech Recognition (ASR) benchmarking:

1. Synthetic Audio Generation: The model synthesizes targeted test sentences containing dense sets of currencies, timestamps, complex fractions, and dates.
2. ASR Decoding: A standardized, high-accuracy ASR engine (such as Whisper or Conformer) decodes the audio into text using explicit verbalized-output settings (disabling ASR post-processing inverse normalization).
3. Targeted Word Error Rate (WER): Accuracy is calculated specifically over the semiotic target intervals to evaluate whether the audio correctly realized the written numerical tokens.

Key Metrics for Verbalization Accuracy

TTS normalization accuracy is assessed using specific performance indicators:

• Token-Level Accuracy: The percentage of semiotic tokens converted exactly to their correct phonetic/orthographic target.
• Class-Specific Error Rate: Error rates broken down strictly by class (e.g., checking DATE accuracy independently from CARDINAL accuracy to pinpoint model blind spots).
• Sentence-Level Exact Match (S-EM): A strict evaluation metric where an entire utterance is considered incorrect if a single number, date, or currency string is misverbalized.