Part IV · Ch. 14 — Unified Multimodal
Connectionist · Text+image+audio+video · Transformer + unified tokens. A single model that perceives and generates across multiple modalities — the frontier direction. Card:
../02-types-of-ai/02-connectionist.kmd.
🎨 Figure
F-IV.14.0— One brain, many senses. Brief: a central Transformer core with "senses" coming in (eye/image, ear/audio, text, video) and generated outputs in the same modalities; a common latent space at the center.
1. Definition and short history
Fuses modalities into a common representation space. Lineage: CLIP (text-image alignment) → vision-language models → multimodal natives (GPT-4o, Gemini, Claude with vision) and unified tokens (Chameleon, Emu3). See Part III, era 7.
2. Foundations
- Linear algebra — shared latent space; cross-attention.
- Information theory — (contrastive) alignment between modalities.
- Linguistics + optics + acoustics — each modality brings its own science.
- Learning theory — transfer across modalities; emergence.
3. Algorithms and architectures
- Per-modality encoders — ViT (image), audio codec/encoder,
text tokenizer → common space.
- Fusion — cross-attention or unified tokens (everything becomes a token).
- Any-to-any generation — output in any modality (text, image, audio).
- Training — contrastive alignment + joint generative objective.
4. Inputs
- Hardware: GPU/TPU (expensive training; many modalities).
- Data: aligned pairs/triples (text-image-audio-video) at scale.
- Data structures: unified tokens, multimodal embeddings, KV-cache.
- Systems: multimodal pipelines; serving with multiple front-ends.
5. Specialized lifecycle
| Stage | Specialization |
|---|---|
| 0 Problem | Which input/output modalities; target capabilities |
| 1 Data | Data aligned across modalities; deduplication; balancing |
| 2 EDA | Coverage per modality, alignment, biases |
| 3 Modeling | Encoders + fusion vs unified tokens; base architecture (LLM) |
| 4 Training | Multimodal pre-training + post-training (instruction, RLHF) |
| 5 Evaluation | Per-modality + cross benchmarks; visual hallucination |
| 5.5 Homologation | Safety across all modalities, multimodal red team |
| 6 Production | Serving with mixed inputs; cost per modality; tools |
| 7 Monitoring | Quality per modality, drift, abuse |
| 8 Retraining | New modalities/capabilities |
| 9 Governance | Privacy (image/voice), deepfakes, multimodal bias |
6. Capabilities, modes and modalities
Multiple kinds of intelligence in a single system: linguistic, visual, auditory; describe images, answer questions about video, converse by voice, generate media. The basis of multimodal agents (ch. 30 — Agent).
7. Limits, risks and ethics
Visual hallucination; cost; biases summed across several modalities; privacy of image/voice; deepfakes. Alignment must cover all modalities.
8. State of the art and examples
GPT-4o, Gemini, Claude (vision), Chameleon, Emu3; convergence with media generation (diffusion/video) and with agency — the type that comes closest to a practical "generalist" AI system.