Unified Multimodal Architectures

Models that process vision, language and audio in a single architecture. Updated in April 2026.

Overview

Unified multimodal architectures treat all inputs (text, image, audio, video) as token sequences in a single representation space. Instead of having a separate language model + vision model + audio model, a single transformer processes everything.

This brings three fundamental advantages:

1. Cross-modal transfer — knowledge learned in one domain (e.g., image captioning) helps in another (e.g., text generation)
2. Native alignment — no extra step needed to "align" representations from different modalities
3. Emergence — capabilities that do not exist in unimodal models arise when all data is trained together

Main Architectures

Flamingo (DeepMind, 2022)

The first model to demonstrate that a pre-trained LLM can learn vision from a few examples (few-shot) without heavy fine-tuning.

Why it matters: It proved you do not need to train from scratch — a pre-trained language LLM can "gain eyes" with relatively light training of the visual adapter.

Chameleon (Meta, 2024)

The first truly "mixed" model: text tokens and image tokens live in the same vocabulary, enabling native interleaved generation (text-image-text-image).

Architecture:

Input → Tokenizer (text) + VQGAN (image) → Unified token space
                                             ↓
                                        Transformer
                                             ↓
                                      Text head + Image head

Limitation: Image tokenization via VQGAN loses resolution — generated images are of moderate quality. But the ability to reason about images and text in the same space is revolutionary.

LLaVA-NeXT (Large Language and Vision Assistant, 2024–2026)

Evolution of the original LLaVA, which connected a visual encoder (CLIP) to an LLM (Llama/Vicuna) via a simple MLP projector.

AnyRes (dynamic resolution): Instead of resizing the image to a fixed size (e.g., 336×336), AnyRes splits the image into patches and processes each one separately, then aggregates. This allows understanding high-resolution images without computational blow-up.

For Kode: LLaVA-NeXT with Qwen2.5-Coder as the base LLM is the strongest candidate for a code VLM — able to understand IDE screenshots, diagrams, and visual code.

InternVL2 / InternVL2.5 (Shanghai AI Lab, 2024–2025)

Architecture: SigLIP (vision encoder) + MLP projector + Qwen2/InternLM2 (LLM). Trained on 10M+ image-text pairs with high-quality data (manual curation + filtering).

Strong point: OCR and document understanding — InternVL2.5 surpasses GPT-4o in DocVQA and ChartQA on several sub-tasks.

Qwen2.5-VL (Alibaba, 2025)

Highlights:

• Dynamic resolution: processes images at native resolution without cropping
• Multilingual OCR: understands text in 30+ languages within images
• Video understanding: processes up to 20 minutes of video with temporal pooling
• GUI agent: trained to interact with graphical interfaces (clicks, types, navigates)

For Kode: Qwen2.5-VL-7B is the best open VLM for code at the moment. It integrates well with the Qwen ecosystem and has a permissive license.

Gemini 1.5/2.0 (Google, 2024–2026)

A proprietary model, but an architectural reference.

Architecture: Transformer with native MoE, unified multimodal tokenization (text → subwords, image → patches, audio → frames, video → temporal frames). All tokens live in the same embedding space.

Architectural lesson: Gemini's key is joint multimodal pre-training — it is not "LLM + visual adapter," it is a single model trained on text + image + audio + video from the start.

PaLI (Pathways Language and Image, Google, 2022–2023)

Approach: Uses T5 as the base and adds a ViT as the visual encoder, with a "bridge" that projects visual features into T5's embedding space.

Contribution: Demonstrated that scaling works for multimodal the same way it works for pure text — more data + more parameters = consistent improvement.

Meta Chameleon vs. Llama 3.2 Vision

For Kode: Llama 3.2 Vision (11B and 90B) is more practical because it has a permissive license and is already optimized for deployment. Chameleon is more interesting as an architectural reference.

Comparison of Approaches

Multimodal Training — Typical Pipeline

Phase 1: Pre-train the vision encoder
  → CLIP/SigLIP on image-text pairs (400M–4B pairs)

Phase 2: Pre-train the LLM
  → Pure text (1T–10T tokens)

Phase 3: Alignment projector
  → Freeze vision encoder + LLM, train only the projector
  → Data: 10M–100M image-text pairs

Phase 4: Multimodal instruction tuning
  → SFT with visual instructions (VQA, captioning, reasoning)
  → ~500K–2M examples

Phase 5: Preference optimization (optional)
  → DPO/RLHF with visual rewards

Estimated cost for a 7B VLM:

• Phase 3: ~50 GPU-hours A100
• Phase 4: ~200 GPU-hours A100
• Phase 5: ~100 GPU-hours A100
• Total: ~350 A100-hours ≈ R$ 50–150K (depends on the provider)

Multimodal Datasets

For Kode

Architecture recommendation

For a Koder-owned VLM:

1. Base LLM: Qwen2.5-Coder-7B or Llama-3.1-8B (already optimized for code)
2. Vision encoder: SigLIP-SoViT-400M (open, good resolution, efficient)
3. Projector: 2-layer MLP (simple, works)
4. Instruction tuning: 500K examples focused on visual code (IDE screenshots, diagrams, UML, flowcharts)

Why not Chameleon-style? Image tokenization via VQGAN loses details critical for code (symbols, visual indentation, syntax highlighting colors). The separate projector + encoder approach preserves more visual information.

Priority datasets for Kode

• IDE screenshots with corresponding code
• UML/ERD diagrams with textual descriptions
• Terminal output with corresponding commands
• Visual code diffs (before/after)
• Technical documentation with figures and text

Minimum hardware

Papers and References