Part IV · Ch. 09 — Music Model
Connectionist · Music · Autoregressive / diffusion over audio tokens. Composes and generates music from text or conditioning. Card:
../02-types-of-ai/02-connectionist.kmd.
🎨 Figure
F-IV.9.0— From prompt to song. Brief: text ("sad lo-fi piano") → audio tokens → waveform with musical structure (bars, layered instruments).
1. Definition and short history
Generates musical audio (instrumental and vocal) conditioned on text, melody or style. Lineage: symbolic MIDI → Jukebox (2020, raw audio) → MusicGen, Suno, Udio (2023+, production quality).
2. Foundations
- Music theory — harmony, rhythm, form, timbre as priors.
- Acoustics / DSP — audio representation, neural codecs.
- Probability — autoregressive/diffusion generation over tokens.
- Information theory — audio compression (codec) defines the generated unit.
3. Algorithms and architectures
- Audio tokenization — neural codec (EnCodec) → discrete tokens.
- Model — autoregressive Transformer (MusicGen) or diffusion over latents.
- Conditioning — text, melody, tempo, instrumentation.
- Multi-track — generation of separate stems (vocals, drums, bass).
4. Inputs
- Hardware: GPU; generating minutes of audio is costly.
- Data: music + metadata/lyrics; licensing is critical.
- Data structures: codec tokens, genre/style embeddings.
- Systems: audio pipelines; source separation.
5. Specialized life cycle
| Stage | Specialization |
|---|---|
| 0 Problem | Generate a track/song; control (genre, BPM, instruments, vocal?) |
| 1 Data | Licensed musical audio, captions, lyrics; cleaning |
| 2 EDA | Genre distribution, quality, duplicates |
| 3 Modeling | Codec + Transformer/diffusion; conditioning; multi-stem |
| 4 Training | Codec + generator training; long |
| 5 Evaluation | Musical quality (human), prompt alignment, originality |
| 5.5 Acceptance | Plagiarism/copy filter, watermark, rights check |
| 6 Production | Conditioned generation; editable stems |
| 7 Monitoring | Quality, rights complaints, abuse |
| 8 Retraining | New genres/styles |
| 9 Governance | Copyright (training and output), royalties, artist voice |
6. Capabilities, modes and modalities
Musical/artistic: composition, soundtracks, jingles, demos; synthetic vocals; accompaniment. Conditioned creativity ([metaphor]).
7. Limits, risks and ethics
Copyright (training data and output similarity); imitation of artists; displacement of musicians; royalties. Watermarking and data curation are central.
8. State of the art and examples
Suno, Udio, MusicGen, Stable Audio; trend: fine control (melody/lyrics), stems, studio quality, and ongoing legal disputes.