Alternatives to the Transformer

Researched as alternatives for efficiency in long context, faster inference and lower memory consumption.

---

State Space Models (SSMs)

Mamba / Mamba-2

• arXiv: 2312.00752 | ICLR 2024
• Authors: Albert Gu & Tri Dao
• Mechanism: Selective SSM — state parameters are a function of the input (selectivity); linear in sequence time
• Advantages vs Transformer:
  • O(n) in memory vs O(n²) for attention
  • 5× higher inference throughput
  • Scales to sequences of 1M+ tokens
• Performance: Mamba-3B surpasses 3B Transformers; matches 6B Transformers on some metrics
• Models that use it: Jamba (AI21 Labs — Mamba+Transformer hybrid), Zamba
• GitHub: github.comstate-spacesmamba

S4 (Structured State Spaces)

• arXiv: 2111.00396
• Predecessor of Mamba: the first high-performance SSM for long sequences
• Improvement: Mamba added the selectivity that S4 lacked

---

Modern Recurrent Networks

xLSTM — Extended Long Short-Term Memory

• arXiv: 2405.04517 | NeurIPS 2024 Spotlight
• Authors: Maximilian Beck et al. (ELLIS Institute)
• Innovations:
  • sLSTM: scalar memory with memory mixing and exponential gating
  • mLSTM: fully parallelizable matrix memory with covariance updates
• Performance: Competitive with Transformers and SSMs in scaling

RWKV — Receptance Weighted Key Value

• arXiv: 2305.13048 | EMNLP 2023
• Authors: Peng et al.
• Mechanism: Can be formulated as a Transformer (parallel training) or an RNN (O(1) inference)
• Advantage: Constant-cost inference without a KV cache; fixed memory
• Performance: RWKV-14B competes with Transformer-14B

---

Hybrid and Linear Attention Mechanisms

RetNet — Retentive Network

• arXiv: 2307.08621
• Microsoft Research
• Mechanism: Retention — three paradigms: parallel (training), recurrent (O(1) inference), chunkwise (long sequences O(n))
• Advantage: Training parallelism + low-cost inference + good performance

Infini-Attention

• arXiv: 2404.07143
• Google
• Mechanism: Compressive memory + masked local attention + linear attention
• Compression: 114× memory reduction
• Result: A 1B LLM scales to 1M context; 8B SOTA in summarizing 500K books

Linear Attention

• Concept: Approximates softmax attention with O(n) complexity using kernel tricks
• Variants: Performer, FNet, cosFormer, GLA (Gated Linear Attention)

---

Liquid Neural Networks

A family of architectures inspired by continuous dynamical systems (ODEs) — neurons with continuous-time nonlinear dynamics, rather than discretized as in RNN/Transformer.

Liquid Time-Constant Networks (LTC)

• arXiv: 2006.04439 | AAAI 2021 | Hasani, Lechner et al. (MIT CSAIL)
• Mechanism: ODEs with learnable time constants — each neuron is a function of the previous state + input via a continuous integrator.
• Emblematic result: 19 neurons piloted a drone in a visual lane-following task where CNN+LSTM needed millions of params.

Closed-form Continuous-time Networks (CfC)

• Nature Machine Intelligence 4, 992-1003 (2022) | Hasani et al.
• Advantage over LTC: closed form of the ODE — dispenses with a numerical solver, 10-100× faster in training and inference while maintaining expressiveness.

Liquid Foundation Models (LFM)

• Liquid AI, 2024-2026 — commercialization of the academic ideas at foundation scale.
• LFM-1B / LFM-3B / LFM-40B (MoE) — competitive with Llama/Mistral in accuracy with constant memory during long inference.
• LFM2 (2026) — second generation focused on edge; see the "Hybrids" entry below.
• Site: liquid.ai

For Kode

• Promising for edge (mobile/desktop on-device); revisit when koder_kit needs models dedicated to small tasks with sub-second latency.

---

Joint Embedding Predictive Architectures (JEPA)

A non-generative self-supervised learning paradigm: it predicts embeddings of masked parts in latent space, not pixels/tokens. Conceptual summary here; deep dive in [[alternative-paradigms]] under "JEPA".

I-JEPA (Image-JEPA)

• arXiv: 2301.08243 | CVPR 2023 | Meta AI (LeCun group)
• Compared with MAE: similar accuracy on ImageNet linear probe with less compute.

V-JEPA / V-JEPA 2

• V-JEPA: Bardes et al., 2024 — video SSL.
• V-JEPA 2: Meta 2025 — 2M+ hours; zero-shot transfer to robotic control.

Why it is here

JEPA is considered by LeCun to be the architectural basis of future world models that would replace auto-regressive LLMs as the path to AGI/AMI. It does not compete with the Transformer in language generation; it competes in representation for perception + action.

For Kode

• If the Stack gains its own vision encoder (Eye 2.0, screen understanding), JEPA is a candidate for more efficient pretraining than CLIP/SigLIP.

---

Language Diffusion Models (dLLMs)

A non-autoregressive generation paradigm: instead of predicting 1 token at a time from left to right (AR), the model starts from a masked/noisy sequence and refines it in parallel by blocks through discrete diffusion steps (discrete denoising) until it converges on the final text. It trades sequential latency (O(n) steps) for a fixed, small number of refinement steps — a large throughput gain, at the cost of some quality vs the equivalent AR.

DiffusionGemma (Google DeepMind, 06102026)

• What: the Gemma family's first open-weights language diffusion model — experimental, an exploration of text diffusion at scale. Weights google/diffusiongemma-26B-A4B-it on Hugging Face.
• Architecture: 26B total / 4B active MoE (8 of 128 experts active), built on the Gemma 4 architecture, but with a generation head via discrete diffusion instead of AR decoding. Generates blocks of 256 tokens in parallel.
• Multimodal: text + image + video input → text output. Context 256K; 140+ languages.
• Speed: 1,000+ tokens/s on an NVIDIA H100, 700+ tokens/s on an RTX 5090 — Google reports up to 4× faster than the equivalent autoregressive Gemma 4.
• Footprint: fits in 18 GB of VRAM quantized (high-end consumer GPU). Native integration: MLX, vLLM, Hugging Face, Unsloth, with optimized NVIDIA NVFP4 kernels.
• Sweet spot: non-linear workflows — code infilling, inline editing, real-time auto-correction (block refinement allows rewriting any position, not just appending).
• Trade-off (stated by Google itself): output quality inferior to standard Gemma 4; for production work at maximum quality the recommendation remains the autoregressive Gemma 4. Positioned as a low-latency / high-throughput tool, not a general replacement.
• License: Apache 2.0.
• Links: ai.google.devgemmadocsdiffusiongemma · deepmind.googlemodelsgemmadiffusiongemma · huggingface.cogooglediffusiongemma-26B-A4B-it

Lineage of dLLMs

• LLaDA (2025) — Large Language Diffusion with mAsking; the first open 8B dLLM competitive with LLaMA-3 8B, a proof of concept that diffusion scales in language.
• Mercury (Inception Labs, 2025) — a commercial dLLM focused on code; the first to commercialize the throughput advantage (≈1000+ tok/s) for coding assistants.
• Gemini Diffusion (Google, I/O 2025 demo) — internal prototype that preceded DiffusionGemma; showed the viability of text diffusion at Google scale.

For Kode

• Real use case: the editor's autocomplete / infill / "ghost text" and real-time auto-correction are latency-bound, not quality-bound — exactly the regime where a dLLM beats AR. A candidate for an inline completions engine (separate from the AR chat/reasoning model).
• Edge: the 18 GB quantized + MLX/vLLM integration puts DiffusionGemma within reach of a dev workstation; revisit when koder_kit needs ultra-low-latency on-device completions.
• Do not use it as the main reasoning/agent model — Google itself signals quality below the AR Gemma 4.

---

Hybrid Architectures

Jamba (AI21 Labs, 2024)

• Combines: Transformer (self-attention) + Mamba (SSM) + MoE
• Advantage: Mamba's better KV cache efficiency with the Transformer's expressiveness

Zamba (Zyphra, 2024)

• Combines: Mamba + periodic attention layers
• Sizes: 2.7B, 7B — competitive for small models

OLMo Hybrid (Allen AI, 2026)

• Combines: Transformer attention layers + Linear RNN layers (RWKV-style)
• Result: Same accuracy as OLMo 3 using 49% fewer pre-training tokens (2× data efficiency)
• First SOTA trained on B200s (Lambda infra)
• Paper: allenai.orgpapersolmo-hybrid

LFM2-24B-A2B (Liquid AI, 2026)

• Combines: Liquid Foundation Model (LFM) + linear attention
• Focus: Edge deployment and on-device inference
• Result: Addresses the "scaling bottlenecks" of traditional LLMs on limited hardware

---

Multi-Memory Architectures

Google Titans (2025)

• Mechanism: 3 types of memory in a single model:
  • Short-term memory: Local attention over the immediate context window
  • Long-term memory: Neural memory module — learns to compress and retrieve information from past contexts
  • Persistent memory: Model parameters (fixed knowledge from training)
• Context: Scales beyond 2M tokens with linear cost
• Differentiator: Long-term memory is learned during fine-tuning, not heuristic

DeepSeek Engram (2026)

• arXiv: 2601.07372 · Code: github.comdeepseek-aiEngram (Apache 2.0)
• Premise: MoE scales capacity via conditional computation; Engram introduces conditional memory as a new axis of sparsity — complementary to (not a replacement for) MoE.
• Mechanism: Massive, static N-gram embedding tables injected into Transformer layers. For each position, a hash of 2-3 token sequences → O(1) lookup in the table. A modernization of the classic N-gram embedding.
• Sparsity Allocation Problem: A U-shaped scaling law that governs the trade-off between neural computation (MoE) and static memory (Engram). Empirical optimum: 75-80% compute + 20-25% memory.
• Results (27B, iso-params + iso-FLOPs vs MoE baseline): MMLU +3.4, CMMLU +4.0, BBH +5.0, ARC-Challenge +3.7, HumanEval +3.0, MATH +2.4. Multi-Query NIAH long-context: 84.2 → 97.0 (delegating local dependencies to the lookup frees attention for global context).
• Differentiator vs RAG: Integrated parametric memory (not external retrieval); "lookup vs compute" decisions are end-to-end trainable.
• Status in V4: NOT integrated into DeepSeek-V4 (Jan2026 paper vs Apr2026 V4 paper). Parallel work from DeepSeek; promotional videos have conflated the two.

---

New MoE Architectures

LatentMoE (NVIDIA, 2026)

• Introduced in: Nemotron 3 Super
• Mechanism: Expert weights are projected into a shared latent space — they are not independent weights
• Result: Better accuracy per parameter AND per FLOP than regular MoEs
• Effect: A smaller model with the capacity of a larger MoE; 2.2–7.5× higher inference throughput

---

Architecture Comparison

---

Note on Relevance for Kode

The frontier models in 2026 are all Transformers or MoE-Transformers. Alternative architectures are interesting for:

• Contexts >1M tokens: Titans or OLMo Hybrid (linear scaling)
• Limited hardware: LFM2/Liquid, RWKV (O(1) inference)
• Training efficiency: Hybrids with linear RNN (2× data efficiency)
• Generation latency (infill/completions): dLLMs (DiffusionGemma) — generation by parallel blocks, ~4× throughput vs AR at the same size

For Kode v1: use a standard Transformer/MoE. Revisit hybrids when there is a need for whole-repository context (>500K tokens).

---

See also

• [[alternative-paradigms]] — neuro-symbolic, Tsetlin, HDC, Forward-Forward, EBM, Active Inference, JEPA deep dive
• [[..06-hardwareneuromorphic]] — Loihi 2, NorthPole, SpiNNaker; SNN training (pairs with Forward-Forward and Predictive Coding)