AI in Audio, Speech, and Music

Speech Recognition (ASR)

Whisper (OpenAI, 2022–2024)

• arXiv: 2212.04356
• Data: 680K hours of supervised audio from the internet
• Capability: 100 languages; transcription + translation; robust zero-shot
• Variants:

• Implementations: openai-whisper, faster-whisper (CTranslate2, 4× speedup), WhisperX (diarization)

---

Canary (NVIDIA, 2024)

• Model: CTC/RNN-T; 1B parameters
• Highlight: Surpasses Whisper Large v3 in English; 4 languages (ENDEES/FR)
• Speed: Faster than Whisper Large

SeamlessM4T v2 (Meta, 2023)

• arXiv: 2312.05187
• Capability: ASR + S2T + S2S + T2TT across 101 languages
• Highlight: Multilingual speech-to-text and speech-to-speech translation

---

Speech Representations (Self-Supervised)

Wav2Vec 2.0 (Meta, 2020)

• arXiv: 2006.11477
• Mechanism: Contrastive learning over audio features without labels
• Impact: Fine-tuning with 10 min of labeled data → good ASR
• Current use: Foundation for speech models in low-resource languages

HuBERT (Meta, 2021)

• arXiv: 2106.07447
• Mechanism: Prediction of audio feature clusters (inspired by BERT)
• Features: Excellent for TTS and voice conversion

SpeechBrain

• URL: speechbrain.github.io
• Toolkit: Open-source for ASR, TTS, speaker recognition, VAD, enhancement

---

Neural Audio Codecs

EnCodec (Meta, 2022)

• arXiv: 2210.13438
• Mechanism: Residual Vector Quantization (RVQ); compresses audio into discrete tokens
• Bitrates: 1.5, 3, 6, 12, 24 kbps
• Use: Foundation for audio language models (AudioLM, MusicGen, VALL-E)

Mimi (Kyutai, 2024)

• Moshi project: Mimi is the codec of the real-time conversation system
• Advantage: Low latency; lower bitrate than EnCodec with similar quality

DAC — Descript Audio Codec (2023)

• arXiv: 2306.06546
• Advantage: Better quality than EnCodec on music and sound effects

---

Voice Synthesis (TTS)

F5-TTS (2024)

• arXiv: 2410.06885
• Mechanism: Flow matching + Transformer (no autoregressive)
• Voice cloning: Zero-shot with 3–5s of reference
• Quality: State of the art open-source in naturalness

XTTS v2 (Coqui, 2023)

• Open-source: Yes
• Languages: 17 languages; zero-shot voice cloning
• Use: Leading open-source multilingual TTS in quality

Kokoro (2024)

• Parameters: 82M — extremely small
• Quality: Comparable to models 10× larger in English
• License: Apache 2.0

Fish Speech (Fish Audio, 2024)

• Languages: ENZHJP + others
• Mechanism: Codec language model over VQVAE tokens
• Cloning: Zero-shot with 5s of reference

VALL-E (Microsoft, 2023)

• arXiv: 2301.02111
• Mechanism: Language model over EnCodec tokens
• Highlight: Voice cloning in 3 seconds (first paper to do it convincingly)

ElevenLabs

• Proprietary: Commercial leader in voice cloning and expressive TTS
• Quality: Best commercial quality in 2024

---

Real-Time Conversation (End-to-End)

Moshi (Kyutai, 2024)

• arXiv: 2410.00037
• Paradigm: First fully end-to-end speech language model
• Mechanism: Processes and generates audio directly (no intermediate transcription)
• Latency: ~200ms (vs 1000ms+ in pipeline systems)
• Dual stream: User speech and model speech in parallel

GPT-4o Realtime API (OpenAI, 2024)

• Mechanism: Audio → GPT-4o → Audio directly (no Whisper/TTS)
• Latency: ~300ms
• Capability: Intonation, interruption, barge-in

Gemini Live (Google, 2024–2025)

• Multimodal: Voice + video in real time
• Integration: Android, Google Workspace

Ultravox (Fixie AI, 2024)

• Architecture: Whisper encoder + Mistral 7B; end-to-end speech-to-speech
• Open-source: Yes; open-source reference for real-time voice

---

Music Generation

MusicGen (Meta, 2023)

• arXiv: 2306.05284
• Mechanism: Language model over EnCodec tokens conditioned by text + melody
• Sizes: 300M, 1.5B, 3.3B
• Open-source: Apache 2.0

AudioLDM 2 (2023)

• arXiv: 2308.05734
• Mechanism: Diffusion in the audio latent space
• Covers: Music, sound effects, speech

Stable Audio (Stability AI, 2024)

• Mechanism: Diffusion with duration conditioning
• Result: High-fidelity musical audio (44.1 kHz stereo)

Suno AI / Udio (2024)

• Proprietary: Generation of complete songs (voice + instruments) from text
• Impact: Democratized music creation; sparked debates about copyright

---

Voice Activity Detection (VAD)

Silero VAD

• Use: Detects the presence of speech in audio in real time
• Integration: faster-whisper, Whisper.cpp, ASR pipelines

---

Audio Benchmarks