Mechanistic Interpretability and Transparency

Why Interpretability?

• Trust: Understand WHY the model gives an answer, not just WHAT it answers
• Alignment: Detect problematic behaviors before they manifest
• Debugging: Locate where errors arise in processing
• Science: Understand what models actually "learned" about the world

---

Mechanistic Interpretability (Mech Interp)

A subfield that attempts to reverse-engineer neural networks — to discover the algorithms implemented by the weights.

Pioneers

• Anthropic: Largest mech interp team; Chris Olah founded the field
• DeepMind: Dedicated team; work on activations and representations
• MIT CSAIL: Discoveries about circuits and features (breakthrough 2026)

---

Fundamental Concepts

Features (Representations)

• Hypothesis: Neural networks represent concepts as directions in activation space
• Feature = direction in activation space that activates for a specific concept
• Linear representation hypothesis: Concepts are represented linearly (sum of vectors)

Superposition

• Problem: Networks with N neurons represent far more than N features (overlap)
• Mechanism: Features share neurons — each neuron responds to multiple features
• Toy model: "Toy Models of Superposition" (Elhage et al., Anthropic, 2022) demonstrated this formally

Circuits

• Concept: Subgraphs of weights that implement specific computational algorithms
• Example: "Induction heads" — an in-context copy mechanism discovered in all Transformers
• Localization: Which attention heads and MLPs are responsible for which behavior?

---

Sparse Autoencoders (SAEs)

The most important technique currently for extracting interpretable features.

How It Works

1. Train an autoencoder with sparse activation over the model's activations
2. The encoder extracts features (many — more than neurons)
3. Each feature corresponds to an interpretable concept

Anthropic — "Scaling and Evaluating SAEs" (2024)

• arXiv: 2408.05147
• Scale: SAEs with 1M+ features in Claude 3 Sonnet
• Result: Discovered features for: Barack Obama, embryo, cancer, violence, cryptography...
• "Brain microscope": Each feature has an automatic interpreter via LLM

Gemma Scope 2 (Google, 2025)

• Release: 2025
• Scale: SAEs for Gemma 2 2B, 9B, 27B — public weights
• Features: 16K to 1M features per layer
• Open-source: SAE weights available on HuggingFace
• Usage: Community research on interpretability

EleutherAI — SAE for Pythia / GPT-NeoX

• Reproduction of SAE techniques on smaller, fully open models

---

Discoveries in Circuits

Induction Heads (Anthropic, 2022)

• arXiv: 2209.11895
• Discovery: A "pattern copying" mechanism that emerges in any Transformer of 2+ layers
• Function: [A][B]...[A] → predicts [B]; basis of in-context learning

Indirect Object Identification Circuit (MIT, 2022)

• arXiv: 2211.00593
• Example: "When Mary and John went to the store, John gave a drink to" → Mary
• Result: A circuit of 26 attention heads identified with specific functions

Factual Associations (MIT, 2022)

• ROME paper: arXiv:2202.05262
• Finding: Facts are stored in specific MLP layers (medial layers)
• Implication: Surgical "editing" of facts in the model (model editing)

---

Model Editing

Modify specific facts in the model without retraining.

ROME — Rank-One Model Editing

• arXiv: 2202.05262
• Mechanism: Identifies and directly modifies the MLP entries that store the fact

MEMIT — Mass-Editing Memory in a Transformer

• arXiv: 2210.07229
• Scale: Edits thousands of facts at once

WISE / GRACE

• Alternative approaches with "external storage" of edits

---

Probing

The simplest technique: train a linear classifier over activations to detect concepts.

• Concept: If a linear probe can predict X from activation Y, then Y represents X linearly
• Usage: Check whether models have a representation of truth/falsehood, sentiment, etc.
• Limitation: A probe working does not prove that the model uses that information

---

Attention Analysis

BertViz / TransformerLens

• BertViz: Visualization of attention patterns (bertviz.org)
• TransformerLens: Mech Interp toolkit (Neel Nanda, Anthropic/DeepMind)
  • URL: github.comneelnanda-ioTransformerLens
  • Features: hooks for activations, logit lens, attention patterns

---

MIT Breakthrough 2026

• Announcement: MIT CSAIL, March 2026
• MIT Tech Review: Named one of the "10 Breakthrough Technologies 2026"
• Discovery: Mapping of "high-level concepts" to specific circuits in large Transformers (70B+)
• Method: SAEs + automated ablation studies at scale
• Result: First confirmed causal mapping (not merely correlational) between concept and circuit
• Implication: Enables surgically "switching off" specific behaviors

---

Emotion Vectors in Claude (Anthropic, 2026)

• Discovery: Researchers identified 12 distinct emotion vectors in Claude's internal activations
• Mapped emotions: Happy, Hostile, Afraid, Blissful, and 8 more affective states
• Method: Linear probing + steering analysis on residual stream activations
• Implication: Language models develop internal representations that behave functionally like emotions — not metaphor, but real computational structure
• Caveat: Existence of a representation ≠ "feeling" emotions — an open philosophical debate

---

Tools and Resources

---

Current Limitations

1. Scale: Techniques work well on GPT-2 (124M); much harder on 70B+
2. Completeness: SAEs capture features but not all behavior
3. Causality: Correlation ≠ causation; many findings are correlational
4. Compositionality: Individually interpretable features ≠ interpretable reasoning

---

Relevance for Kode

• Debugging behaviors: If Kode refuses legitimate code or accepts malicious code, mech interp can locate the error
• Trust: For use in critical environments, interpretability is a requirement
• Recommendation: Integrate TransformerLens into the evaluation pipeline; monitor Gemma Scope and Anthropic releases for applicable techniques