AI in Video, 3D, and World Models

Video Generation

Sora (OpenAI, 2024–2025)

• Release: February 2024 (demo); December 2024 (access)
• Architecture: Video Diffusion Transformer (spacetime DiT)
• Capability: Videos up to 1 minute; multiple aspect ratios; coherent physics
• Mechanism: "Spacetime patches" — treats frames as 3D tokens
• Sora 2 (2025): Higher resolution; better physics; consistent character generation

Veo 3 / Veo 3.1 (Google DeepMind, 2025)

• Release: Google I/O 2025
• Highlight: Native audio generation alongside video (lip-sound synchronization)
• Quality: Competitive with Sora; more coherent physics in some cases
• Veo 3.1: Higher resolution; realistic sound effects

HunyuanVideo (Tencent, 2024)

• arXiv: 2412.03603
• Open-source: Yes; public weights
• Specifications: 13B parameters; 720p at 5s; best open-source available
• Quality: Close to Sora in most cases

CogVideoX (Zhipu AI / Tsinghua, 2024)

• arXiv: 2408.06072
• Open-source: Yes (Apache 2.0)
• Variants: 2B and 5B parameters

LTX-Video (Lightricks, 2024)

• Open-source: Yes
• Highlight: Very fast generation (a few seconds on A100)
• Size: 2B parameters; good quality for size

Kling (Kuaishou, 2024)

• Access: API; kuaishou.com/keling
• Highlight: Realistic physics of faces and bodies; popular in Asia

Runway Gen-3 Alpha (2024)

• Proprietary: Runway ML
• Highlight: Temporal coherence; consistent characters; virtual camera integration

---

3D Representation

NeRF — Neural Radiance Fields (2020)

• arXiv: 2003.08934 (Mildenhall et al., UC Berkeley)
• Mechanism: MLP that maps (x, y, z, θ, φ) → (color, density); ray marching for render
• Result: Photorealistic 3D reconstruction from multiple images
• Variants: Instant-NGP (1000× faster), NeRF-W (in the wild), Mip-NeRF

3D Gaussian Splatting (2023)

• arXiv: 2308.04079
• Mechanism: Represents the scene as a cloud of 3D Gaussians with color and opacity; rasterization
• Speed: Real-time render (30+ FPS vs NeRF, which is slow)
• Quality: Similar to NeRF; much faster for interactivity
• Impact: New standard for interactive 3D reconstruction

Shap-E (OpenAI, 2023)

• arXiv: 2305.02463
• Mechanism: Diffusion over implicit NeRF parameters
• Input: Text or image → 3D object
• Open-source: Yes

TRELLIS (Microsoft, 2024)

• arXiv: 2412.01506
• Mechanism: 3D Gaussian Splatting + Mesh via flow matching
• Quality: State of the art in 3D generation from text/image

Hunyuan3D-2 (Tencent, 2025)

• Mechanism: Multi-view diffusion + 3D reconstruction
• Open-source: Yes
• Result: High-quality 3D objects in seconds

Zero-1-to-3 / Zero123++ (Columbia, 2023)

• arXiv: 2303.11328
• Mechanism: Given 1 image, generates views from any angle
• Use: Foundation for many 3D reconstruction systems

---

World Models — Simulators of the World with AI

DreamerV3 (DeepMind, 2023)

• arXiv: 2301.04104
• Mechanism: Learns a model of the world → plans and acts within the model (imagination)
• Result: Mastered Minecraft diamond collection without reward shaping; works across 150+ domains
• Architecture: RSSM (Recurrent State Space Model) + attention

Genie 2 (DeepMind, 2024)

• Release: December 2024
• Mechanism: Learns interactive 3D worlds from video
• Capability: Given 1 image, generates a navigable and interactive 3D world
• Resolution: 360p at 30fps for ~1 minute with consistency

GameNGen (Google, 2024)

• arXiv: 2408.14837
• Mechanism: Diffusion model that simulates DOOM in real time (20 FPS)
• Highlight: First real game simulated by a neural network in real time
• Proof of concept: Games as neural networks

DIAMOND / WHAM

• Research: Simulation of Atari games via diffusion
• Result: Reinforcement learning within the learned model

---

Cosmos (NVIDIA, 2025)

• Release: CES 2025 / GTC 2025
• Focus: World Foundation Models for robotics and autonomous vehicles
• Variants: Cosmos-1.0 (1B to 14B parameters)
• Capability: Physically consistent video generation; scene re-rendering
• Open-source: Yes (part of the weights)
• Use: Synthetic data to train robots; simulation of driving scenarios

---

Video Understanding

VideoLLaMA / Video-LLaVA

• Mechanism: Video encoder + LLM for QA and description
• Use: Content analysis, automatic description, semantic search in video

Gemini 2.5 Pro / Video

• Capability: 1M token context → processes videos of 1+ hour
• Result: Precise QA over long videos (documentaries, meetings, lectures)

---

Video Benchmarks

---

Comparative Table of Video Generation (2025)