Part V · 6 — AI for programming

draft

The best domain for a small team to compete with billion-dollar labs — and why. It is the clearest case of the levers in this part working together.

🎨 Figure F-V.6 — Why code wins. Brief: three interlocking gears labeled "verifiable reward", "filterable data", "short feedback", turning a code model at the center. Compendium palette.

Why code wins

6.1 The three unique properties of programming

  1. Verifiable reward — the code runs or it doesn't → enables RLVR at scale.
  2. Abundant and filterable data — GitHub, Stack Overflow, commits, PRs;

    filterable by stars, tests, build rate.

  3. Short feedback loop — the programmer sees the result in seconds → a real

    preference signal almost free.

That is why teams of 5–50 people (Cursor, Cognition, Aider, Continue.dev) compete with billion-dollar labs in the code domain.

6.2 The 7 decisive ingredients

  1. FIM (Fill-in-the-Middle) — reorganize data as prefixsuffixmiddle.
  2. Long-context over the entire repository — long window + repo-level *context

    engineering*.

  3. Execution and error feedback — safe sandbox (Docker, Firejail, E2B, Modal).
  4. RLVR with tests as reward — problems with tests → reward = passed?
  5. Tree search and self-repair at test-time — generate N, execute, iterate.
  6. Tokenizer optimized for code — reduces tokens by 20–30%.
  7. Grammar-constrained decoding — force syntactically valid outputs.

6.3 Code models, datasets, and benchmarks

  • Open models: Qwen2.5-Coder (0.5B–32B), DeepSeek-Coder-V2 (MoE up to 236B),

    DeepSeek-V3R1, Codestral, StarCoder2, Llama 3.x4.x.

  • IDEs/agents: Cursor, Windsurf, Claude Code, Aider, Continue.dev,

    Devin/OpenHands.

  • Datasets: The Stack v2 (~900B tokens), GitHub Archive, CommitPack,

    SWE-bench Train, APPSCodeContestsLiveCodeBench.

  • Benchmarks: SWE-bench Verified (the most important), LiveCodeBench (avoids

    contamination), HumanEval/MBPP (saturated).

6.4 Essential papers

  • FIM — Bavarian et al. 2022 (arXiv 2207.14255).
  • Qwen2.5-Coder — Hui et al. 2024 (arXiv 2409.12186).
  • DeepSeek-Coder-V2 — 2024 (arXiv 2406.11931).
  • Phi-1 (synthetics as foundation) — Gunasekar et al. 2023 (arXiv 2306.11644).
  • SWE-bench — Jimenez et al. 2024 (arXiv 2310.06770).
  • ReAct (the basis of every agent) — Yao et al. 2022 (arXiv 2210.03629).

Concrete application of all of this in a real product: the case study 07-kode-case-study.kmd.