Annotating Emotion Tags in TTS Speech Databases

This article provides an overview of the primary methodologies used to annotate subjective emotional tags in expressive speech databases for Text-to-Speech (TTS) synthesis. Developing natural, expressive synthetic speech requires reliable affective labeling, which presents challenges due to the inherently subjective nature of human emotion. The sections below outline the dominant annotation frameworks, measurement dimensions, annotator workflows, and quality-control techniques used to transform subjective human perception into structured data suitable for training modern neural TTS models.

Categorical Emotion Annotation

The categorical approach assigns discrete emotion labels to speech utterances based on predefined taxonomies.

• Basic Emotion Sets: Most corpora traditionally rely on Paul Ekman’s six primary emotions (happiness, sadness, anger, fear, disgust, and surprise) alongside a neutral baseline.
• Secondary and Domain-Specific Sets: Modern expressive TTS often requires more nuanced tags, such as sarcasm, excitement, boredom, empathy, or professional formality.
• Multi-Label and Intensity Scoring: Because real-world speech rarely exhibits pure, single-category emotions, annotators are frequently asked to assign primary and secondary tags, often accompanied by a Likert scale (e.g., 1 to 5) indicating emotional intensity.

Dimensional Emotion Models

Dimensional modeling avoids discrete boundaries by positioning emotional states along continuous axes. This method provides the fine-grained, continuous control needed for latent emotion embeddings in neural TTS.

• Valence-Arousal-Dominance (VAD):
  • Valence: Measures the positivity or negativity of the emotion.
  • Arousal (Activation): Measures the physiological energy or excitement level conveyed in the voice.
  • Dominance (Power): Measures the degree of control or submissiveness apparent in the speaker’s tone.
• Self-Assessment Manikins (SAM): To make dimensional ratings intuitive for annotators, visual non-verbal pictorial scales are standardly employed during the evaluation process to rate VAD dimensions accurately without linguistic bias.

Utterance-Level vs. Continuous Time-Series Annotation

Annotation methodologies vary based on temporal granularity:

• Utterance-Level Tagging: Annotators listen to an entire audio segment and assign a single summary label or dimensional coordinate. This is standard for sentence-level TTS architectures.
• Continuous Time-Series Tracking: Tools such as GTrace or Feeltrace allow annotators to use a mouse, joystick, or slider to adjust emotional coordinates in real-time as the audio plays. This yields dynamic affective contours, which are useful for modeling intra-utterance emotional variation and prosodic shifts.

Annotator Sourcing and Bias Mitigation

Because emotional perception varies based on culture, gender, and individual context, database curators use distinct sourcing methodologies:

• Expert vs. Crowdsourced Annotators: While trained linguists and psychologists provide consistent, rule-adherent annotations, crowdsourced pools (via platforms like Prolific or MTurk) provide diverse perceptual baselines representative of general listeners.
• Inter-Annotator Agreement: Subjectivity requires robust statistical metrics to gauge consistency. Standard metrics include Cohen’s Kappa (for two raters), Fleiss’ Kappa (for multiple categorical raters), and Krippendorff’s Alpha (which accommodates missing data and ordinal/interval scales).
• Soft Labels and Probability Distributions: Rather than enforcing a forced-choice majority vote, modern TTS workflows increasingly retain the full distribution of annotator votes. These "soft labels" capture genuine emotional ambiguity and prevent the loss of perceptual nuance.

Machine-Assisted and Semi-Supervised Annotation

Given the cost and fatigue associated with human listening tests, hybrid methodologies are common:

• Speech Emotion Recognition (SER) Pre-Tagging: Pre-trained acoustic models (such as wav2vec 2.0 or HuBERT fine-tuned on emotion datasets) generate initial pseudo-labels.
• Human-in-the-Loop Validation: Human annotators verify, reject, or adjust the model's confidence scores, accelerating the labeling process while preserving subjective human oversight.