Part IV · Ch. 07 — ASR (Speech Recognition)

draft

Connectionist · Audio→text · Transformer / CTC. Transcribes speech into text. Card: ../02-types-of-ai/02-connectionist.kmd.

🎨 Figure F-IV.7.0 — From wave to text. Brief: waveform → mel spectrogram → encoder → transcribed text; temporal alignment indicated.

ASR — speech to text

1. Definition and short history

Converts speech audio into text. Lineage: HMM-GMM (classical, see [ch. 20 — HMM]) → DeepSpeech (CTC) → wav2vec 2.0 (self-supervision) → Whisper (2022, robust and multilingual).

2. Foundations

  • Acoustics / psychoacoustics — speech production and perception.
  • Signal processing (DSP) — STFT, mel spectrogram, MFCC.
  • Probability — sequence-to-sequence alignment (CTC, attention).
  • Linguistics — phonetics, language models for rescoring.

3. Algorithms and architectures

  • Front-end — mel spectrogram.
  • Encoder — Transformer/Conformer (conv + attention).
  • Alignment — CTC or encoder-decoder with attention.
  • Self-supervision — wav2vec 2.0 pre-trains on unlabeled audio.
  • Decoding — beam search with an optional language model.

4. Inputs

  • Hardware: GPU (training); inference feasible on edge (small models).
  • Data: audio + transcriptions (many languages, accents, noise).
  • Data structures: spectrogram tensors, hypothesis lattices.
  • Systems: audio pipelines; streaming for real time.

5. Specialized life cycle

Stage Specialization
0 Problem Languages, streaming vs batch, domain (medical/call center), latency
1 Data Audio+transcription, accent/noise diversity; alignment
2 EDA Distribution of languages, SNR, duration, label quality
3 Modeling CTC vs attention; Conformer; size; streaming
4 Training Self-supervised pre-training + fine-tune; augmentation (SpecAugment)
5 Evaluation WER (word error rate), robustness to noise/accent
5.5 Acceptance Tests per accent/language, bias, streaming latency
6 Production Low-latency streaming; punctuation; diarization
7 Monitoring WER in production, audio drift (new accents/channels)
8 Retraining Field data, new languages
9 Governance Voice privacy, consent, accent/gender bias

6. Capabilities, modes and modalities

Auditory→linguistic: transcription, captioning, voice commands, assistant input; the base of voice agents (with TTS, ch. 08).

7. Limits, risks and ethics

Bias toward accents/low-resource languages; noise and overlapping speech; privacy of recordings; voice spoofing.

8. State of the art and examples

Whisper (multilingual, robust), Conformer; trend: unified streaming ASR with translation and with an LLM (audio→LLM directly).