Alternative Paradigms to Dense Deep Learning

Research lines that reject or modify central premises of the mainstream (differentiable neural network + backprop + scale). Relevant when the bottleneck is memory, energy, interpretability or symbolic reasoning — not pure performance on language benchmarks.

Process note: this file is updated by /k-ia-compendium via Layer D (explicit search for novelties in each line listed below).

---

Neuro-Symbolic AI

Combines neural learning (pattern matching, perception) with symbolic reasoning (rules, logic, search). Premise: structured reasoning tasks (mathematics, programming, planning) benefit more from a symbolic layer than from more parameters.

AlphaGeometry / AlphaGeometry 2

• Paper: Nature 625, 476–482 (2024) | DeepMind
• Mechanism: LLM generates "auxiliary constructions" (intuition) + symbolic prover verifies/deduces (rigor). Loop until the proof closes.
• Result: 2530 IMO 2000-2022 problems (gold-medal level) — previous SOTA: 1030.
• AlphaGeometry 2 (2025): domain expansion + Gemini-based LLM; solved IMO 2024 P4.
• Lesson: ~100M neural params + symbolic prover beats a pure 100B+ transformer in olympiad geometry.

AlphaProof

• DeepMind, 2024 — version for algebra/number theory via Lean 4.
• Mechanism: AlphaZero-style RL over the space of proofs in Lean; LLM proposes tactics, solver verifies.
• Result: IMO 2024 silver (4/6 problems).

DreamCoder

• arXiv: 2006.08381 | Ellis, Solar-Lezama (MIT/CSAIL)
• Mechanism: "wake-sleep" alternating program synthesis (search) + abstraction of reusable subprograms (library learning).
• Domains: lists, LOGO graphics, regex, symbolic physics.
• Advantage: learns an explicit library of concepts; each concept is a readable program.

Differentiable Inductive Logic (∂ILP / NS-CL)

• ∂ILP: arXiv:1711.04574 — a Prolog program whose weights are differentiable; learns rules from examples.
• Neuro-Symbolic Concept Learner (Mao et al., ICLR 2019) — VQA with neural perception + symbolic reasoning.
• Logic Tensor Networks (LTNs) — Serafini & d'Avila Garcez, AAAI 2016 — first-order logic formulas with differentiable fuzzy semantics.

For Kode

• Useful for modules where rules are known and fixed (accounting, BR tax rules, ICP-Brasil validation) — neuro-symbolic can deliver high accuracy with a small model + auditability.
• Does not replace a general LLM; complements it on structured tasks.

---

Tsetlin Machines

Learning based on propositional logic (conjunctive clauses) controlled by Tsetlin automata — no gradients, no neural networks.

Original Tsetlin Machine

• arXiv: 1804.01508 | Ole-Christoffer Granmo (Univ. Agder)
• Mechanism: N automata per feature decide to includeexclude each literal in clauses; rewardpunishment via Type I/II feedback.
• Advantages:
  • The model is a set of readable boolean clauses (interpretable by construction)
  • Runs on a microcontroller (kB of RAM, no FPU)
  • CPU training competitive with SVM/Random Forest on tabular data

Recent variants

• Convolutional Tsetlin Machine (arXiv:1905.09688) — competitive with CNN on MNISTFashion-MNISTCIFAR-10.
• Coalesced TM (arXiv:2108.07594) — clause sharing across classes; reduces memory 5-10×.
• Composite TM / Plug-and-Play TM (2024-2026) — combines multiple specialized TMs.
• Graph Tsetlin Machine (2025) — heterogeneous graphs with structural clauses.

Hardware

• Mignon AI (Agder spin-off, 2024) — native Tsetlin chip, sub-mW inference.
• Open FPGA implementations (github.com/cair).

For Kode

• Candidate for edge inference in mobileTVwearable variants when <1ms latency and battery matter more than absolute SOTA.
• Useful for explainable models under LGPD/AI Act — clauses are directly auditable.

---

Hyperdimensional Computing / Vector Symbolic Architectures (HDC/VSA)

Representation as binary/bipolar hypervectors of 10,000+ dimensions; semantics via algebraic operations (bind, bundle, permute) instead of gradient learning.

Foundations

• Kanerva, P. (2009) — Hyperdimensional Computing — the original manifesto
• Tensor Product Representations (Smolensky 1990) — predecessor
• Holographic Reduced Representations (Plate 1995) — HRR variant

Modern frameworks

• Torchhd (Heddes et al., JMLR 2023) — a PyTorch library for HDC; benchmarks on UCI, EuroSAT, ISOLET, EMG.
• OpenHD (UC Irvine) — HDC runtime for CPU/FPGA.

Practical applications

• Classification of biomedical signals (ECG, EMG, EEG) with one-shot or few-shot learning.
• Wearables (Apple/Samsung research papers 2024-2025) — gesture recognition + activity classification on Cortex-M chips.
• Noise-robust associative memory — graceful degradation with 10-30% bit flips.

Advantages vs deep learning

• One-shot learning without fine-tuning
• Arithmetic model (no iterative training); minutes on a CPU vs hours on a GPU
• Inherently parallelizable on custom hardware

For Kode

• Candidate for on-device event detection (touch, gesture, complementary wake-word) when model size must be below 100KB.
• Relevant spec: specs/voice/wake-word.kmd could gain an HDC backend alternative to TFLite.

---

Learning Algorithms Alternative to Backprop

Forward-Forward Algorithm

• Hinton (2022) — The Forward-Forward Algorithm: Some Preliminary Investigations
• Mechanism: replaces forward+backward with two forwards — one with positive data (real), another with negative (synthetic/shuffled); each layer maximizes "goodness" for positives and minimizes it for negatives.
• Advantages:
  • No need to store activations for the backward pass → O(1) memory in depth
  • Maps directly to neuromorphic hardware ([[neuromorphic]])
  • Enables asynchronous layer-by-layer training
• Current limitation: still behind backprop in accuracy; active in research (several extensions 2023-2025).

Predictive Coding (PC)

• Hierarchical Predictive Coding (Rao & Ballard 1999; Friston 2005)
• PC Networks (Whittington & Bogacz 2017; Millidge et al. 2022) — approximate backprop with local Hebbian rules.
• Connection with biology: considered a plausible model of the visual cortex.

Equilibrium Propagation

• Scellier & Bengio (2017) — physical systems relaxing to an energy minimum compute gradients locally.
• Connects with Energy-Based Models and analog computing.

For Kode

• Do not use in production today. Track: if neuromorphic hardware matures, these algorithms become the only viable option for on-chip training.

---

Energy-Based Models (EBMs) and Modern Hopfield

Models defined by an energy function \(E(x)\) — inference = finding the \(x\) that minimizes \(E\).

Modern Hopfield Networks

• Ramsauer et al. (ICLR 2021) — Hopfield Networks Is All You Need (arXiv:2008.02217)
• Result: continuous Hopfield with an exponential energy function has exponential capacity (vs linear for the classic version) and its update rules are equivalent to Transformer attention.
• Theoretical implication: attention = associative memory; opens the door to EBM-based attention.

Modern EBMs (Yann LeCun)

• LeCun has advocated EBM as a unifying framework since 2006; reinforced it in 2022 with A Path Towards Autonomous Machine Intelligence.
• JEPA (next section) is the current expression of Meta's EBM agenda.

Joint Energy Models (JEM)

• Grathwohl et al. (ICLR 2020) — a classifier is also generative via \(p(x,y) \propto e^{-E(x,y)}\).

---

Joint Embedding Predictive Architectures (JEPA)

Learns latent representations by predicting embeddings (not pixels/tokens) — a non-generative approach to self-supervised learning.

I-JEPA (Image-JEPA)

• arXiv: 2301.08243 | Meta / LeCun group
• Mechanism: predict the embedding of masked patches from visible patches, in the latent space of an encoder.
• Result: competitive with MAE/iBOT using much less compute.

V-JEPA / V-JEPA 2

• V-JEPA (Bardes et al., 2024) — video; learns temporal dynamics.
• V-JEPA 2 (Meta, 2025) — trained on 2M+ hours of video; zero-shot transfer to robotic control.

Advantages vs generative paradigms

• Does not waste capacity predicting irrelevant pixels (texture, noise)
• More data-efficient than MAE/MIM
• Focus on representation, not generation — aligned with LeCun's "world model" vision

Connection with world models

See also 09-applications/video-3d-world-models.md (DreamerV3, Genie 2, Cosmos).

For Kode

• Relevant if the Stack ever needs its own vision encoder (Eye 2.0, screen understanding, video moderation) — JEPA is more compute-efficient than CLIP/SigLIP for pre-training from scratch.

---

Active Inference / Free Energy Principle

Karl Friston's (UCL) framework unifying perception, action and learning under a single principle: minimize surprise (variational free energy).

Premise

The agent maintains a generative model of the world; it acts to reduce the discrepancy between predictions and observations. Backprop deep learning is a special case.

Recent practical implementations

• pymdp (Heins et al., JOSS 2022) — a Python library for Active Inference in discrete POMDPs.
• VERSES AI (2023-2026) — a company commercializing Active Inference (Genius platform); claims of radically superior efficiency over classic RL on some benchmarks.
• Deep Active Inference (Çatal, Tschantz et al., 2020-2024) — combines VAE/transformer with Active Inference in the objective.

State in 2026

• Theoretically promising; practical adoption still niche (robotics, computational neuroscience).
• Still without its own "ChatGPT moment."

For Kode

• Not actionable today. Track VERSES and robotics/agents papers — if a use case appears where it beats classic RL with orders of magnitude less data, reassess.

---

Comparison table

---

Consolidated recommendation for Kode

Short term (next 12 months): none of these paradigms replaces the LLM+MoE+Transformer stack. Monitor.

Medium term (12-36 months): if wearable / TV / mobile variants need <1mW or <100KB inference, Tsetlin Machines + HDC enter as serious candidates — possibly via engines/sdk/koder_kit android-side.

Long term / opportunistic: neuro-symbolic is the most consequential bet for the Koder Stack — it aligns with the goal of auditable AI + LGPD/AI Act friendliness in heavily regulated domains (public health, digital forensics, BR accounting).

Track with priority: AlphaGeometry-class systems, VERSES claims, any Tsetlin paper with >90% performance on a standard benchmark.