Part IV · Ch. 10 — Video Model
Connectionist · Video · Spatio-temporal DiT (diffusion). Generates video coherent in time from text/image. Card:
../02-types-of-ai/02-connectionist.kmd.
🎨 Figure
F-IV.10.0— Coherence in time. Brief: sequence of generated frames, with temporal attention arrows linking objects across frames (identity/motion consistency).
1. Definition and short history
Extends diffusion to the temporal dimension, generating coherent clips. Lineage: video GANs → video diffusion → Sora, Veo, Kling, Runway (2024+). It is the most compute-expensive generation modality.
2. Foundations
- Physics / optics — motion, lighting, physical consistency.
- Probability / stochastic processes — diffusion (ch. 02) in time.
- DSP / compression — codecs and spatio-temporal latents.
- Geometry — implicit 3D coherence, camera.
3. Algorithms and architectures
- Spatio-temporal latent — 3D VAE compresses video.
- Spatio-temporal DiT — diffusion with temporal attention + spatial.
- Conditioning — text, initial image, keyframes, camera.
- Coherence — attention across frames; identity/motion consistency.
4. Inputs
- Hardware: heavy GPU/TPU; memory is the bottleneck (long sequences).
- Data: video + captions; licensing and curation are expensive.
- Data structures: 5D tensors (time), spatio-temporal latents.
- Systems: video pipelines; generation by windows/chunks.
5. Specialized life cycle
| Stage | Specialization |
|---|---|
| 0 Problem | Duration, resolution, FPS, control (textimagecamera) |
| 1 Data | Licensed video+text, deduplication, filtering |
| 2 EDA | Scene/motion distribution, quality, biases |
| 3 Modeling | 3D VAE + temporal DiT; conditioning; temporal window |
| 4 Training | Extremely costly; checkpointing; staged training |
| 5 Evaluation | Quality, temporal coherence, prompt alignment, physics |
| 5.5 Acceptance | Deepfake safeguards, watermark (C2PA), content filters |
| 6 Production | Generation by chunks; high cost/latency; queues |
| 7 Monitoring | Quality, cost, abuse (disinformation) |
| 8 Retraining | New styles, longer duration/higher resolution |
| 9 Governance | Disinformation, deepfakes, image rights, consent |
6. Capabilities, modes and modalities
Visualtemporalartistic: clips from text/image, animation, effects, storyboard. Converging with world models (dynamics simulation).
7. Limits, risks and ethics
Very high compute cost; physical/temporal incoherences; video deepfakes and disinformation; image rights and data. Provenance watermarking is critical.
8. State of the art and examples
Sora, Veo, Kling, Runway Gen-3; trend: longer clips, camera control, synchronized audio, and a bridge to world models (related chapter: robotics).