Part IV · Ch. 08 — TTS (Voice Synthesis)
Connectionist · Text→audio · Autoregressive / diffusion + vocoder. Generates natural speech from text. Card:
../02-types-of-ai/02-connectionist.kmd.
🎨 Figure
F-IV.8.0— From text to voice. Brief: text → acoustic model → spectrogram → vocoder → waveform; indicate timbre/emotion control.
1. Definition and short history
Converts text into speech. Lineage: concatenative/parametric (classical) → Tacotron + WaveNet (neural) → VALL-E and neural codecs (zero-shot voice cloning) → diffusion/flow TTS with high naturalness.
2. Foundations
- Acoustics / psychoacoustics — prosody, intonation, timbre.
- DSP — mel spectrogram, vocoding.
- Probability — autoregressive or diffusion generation.
- Linguistics — phonemes, grapheme-to-phoneme, prosody.
3. Algorithms and architectures
- Acoustic model — text/phoneme → spectrogram (Tacotron, FastSpeech) or
→ audio tokens (neural codec).
- Vocoder — spectrogram → waveform (HiFi-GAN, WaveNet).
- Neural codecs — EnCodec/Mimi tokenize audio for LLM-style models.
- Cloning — zero-shot from a few seconds (VALL-E).
4. Inputs
- Hardware: GPU; inference optimizable for real time/edge.
- Data: text-audio pairs with varied speakers; clean audio.
- Data structures: spectrograms, codec tokens, voice embeddings.
- Systems: audio pipelines; streaming for conversation.
5. Specialized life cycle
| Stage | Specialization |
|---|---|
| 0 Problem | Naturalness, languages, voice/emotion control, latency (real time?) |
| 1 Data | Multi-speaker text-audio; aligned transcription; voice consent |
| 2 EDA | Phonetic coverage, recording quality, speaker diversity |
| 3 Modeling | Acoustic (autoregressive/diffusion) + vocoder; neural codec |
| 4 Training | Acoustic + vocoder training; voice fine-tune |
| 5 Evaluation | MOS (naturalness), intelligibility, speaker similarity |
| 5.5 Acceptance | Anti-spoofing, audio watermark, cloning consent |
| 6 Production | Low-latency streaming; prosody control |
| 7 Monitoring | Quality, abuse (voice fraud) |
| 8 Retraining | New voices/languages |
| 9 Governance | Non-consented cloning, fraud, watermark, vocal identity |
6. Capabilities, modes and modalities
Auditory/expressive: narration, assistants, dubbing, accessibility; voice cloning; controllable emotion and style.
7. Limits, risks and ethics
Unauthorized cloning and fraud (voice scams); audio deepfakes; accent bias; the need for watermarking and consent.
8. State of the art and examples
ElevenLabs, VALL-E 2, diffusion TTS; convergence with the LLM (native voice in low-latency conversational assistants).