Semi-Supervised Audio Scraping for Podcast TTS

Semi-supervised data scraping allows developers to transform raw, unstructured podcast audio into clean, high-fidelity voice datasets for Text-to-Speech (TTS) models. By combining automated web scraping with machine learning techniques—specifically Voice Activity Detection, speaker diarization, automated speech-to-text pseudo-labeling, and confidence-based filtering—this pipeline isolates single-speaker speech, removes noise, and aligns transcripts without requiring tedious manual annotation.

1. Automated Harvesting and Pre-Filtering

The extraction pipeline begins by scraping public podcast directories, RSS feeds, or media hosting platforms for audio files. Metadata such as guest lists, episode descriptions, and file formats are scraped simultaneously. To optimize efficiency, the system applies heuristic filters at the ingestion stage, prioritizing high-bitrate, lossless, or uncompressed audio files (e.g., FLAC or high-quality MP3/AAC) while discarding low-bitrate streams that suffer from compression artifacts.

2. Voice Activity Detection and Audio Slicing

Raw podcast files are typically long (30 to 120 minutes) and contain substantial non-speech segments, such as intro music, sound effects, and silent pauses. Voice Activity Detection (VAD) models scan the raw audio to detect human vocal presence, slicing continuous audio into manageable chunks—typically between 2 and 15 seconds. Chunks containing music beds, heavy background noise, or prolonged silence are automatically flagged and discarded.

3. Speaker Diarization and Target Voice Isolation

Most podcasts feature multiple hosts or guests talking simultaneously. Semi-supervised pipelines apply speaker diarization algorithms to answer the question, "who spoke when?" Diarization systems extract acoustic embeddings (such as x-vectors or d-vectors) to cluster speech segments by individual identity. Once the dominant target speaker's cluster is identified, speech segments containing cross-talk, interruptions, or secondary speakers are stripped away, leaving only isolated clips of the target voice.

4. Acoustic Cleaning and Quality Scoring

Before text alignment, the isolated speech clips pass through an audio-enhancement and quality-assurance layer:

• Denoising and Dereverberation: Deep learning models suppress low-level ambient noise, electrical hum, and room acoustics (reverb).
• Signal-to-Noise Ratio (SNR) Metrics: Clips failing a pre-defined SNR threshold are pruned.
• Clipping and Distortion Detection: Clips with digital clipping, distortion, or heavy dynamic range compression are filtered out.

5. Pseudo-Labeling via Automatic Speech Recognition (ASR)

Because scraped data lacks ground-truth transcriptions, the pipeline uses a pre-trained Automatic Speech Recognition (ASR) model (such as Whisper or Conformer) to generate "pseudo-labels." The ASR model transcribes each sliced, single-speaker audio chunk.

6. Confidence Scoring and Text Normalization

To ensure dataset reliability, the generated transcripts undergo semi-supervised validation:

• Confidence Filtering: ASR models output token-level probability scores. If a segment’s average confidence score falls below a set threshold (e.g., 90%), the sample is discarded to prevent training the TTS model on misaligned or hallucinated text.
• Dual-Model Cross-Validation: Some pipelines pass the audio through two independent ASR systems; if the transcriptions differ significantly, the clip is rejected.
• Text Normalization: Remaining transcripts are normalized by expanding numbers, currency, dates, and abbreviations into their full phonetic or written equivalents (e.g., transforming "$5" into "five dollars").

7. Dataset Packaging for TTS Training

The surviving audio-text pairs are formatted into standard TTS training datasets. The audio is resampled to a uniform sampling rate (typically 22.05 kHz, 24 kHz, or 48 kHz), normalized for loudness (typically around -23 LUFS), and mapped into metadata manifests containing relative audio file paths alongside their normalized transcriptions.