Part V · 2 — Data and datasets

draft

Lever #1. "Garbage in, garbage out" is no cliché — it is the difference between model classes. In scientific domains, data is the bottleneck, not compute nor architecture.

🎨 Figure F-V.2 — The data funnel. Brief: large funnel receiving raw data (web, code, sensors) and filtering through layers (deduplication, quality, safety, license) down to a thin jet of "golden data". Compendium palette.

The data funnel

2.1 Ready-made datasets (examples)

  • The Stack v2 — ~900B tokens of code, permissive licenses (bigcode).
  • FineWeb-Edu — web filtered by educational quality (>> raw Common Crawl).
  • SWE-bench Train — real issue/PR pairs.
  • CommitPack / CommitChronicle — diff + commit message.
  • APPS, CodeContests, LiveCodeBench — problems with tests.

2.2 Synthetic data — the real lever

  • Generate problem + solution + tests; filter the ones that pass

    (Phi/Qwen-Coder style).

  • Generate line-by-line explanations (code → doc).
  • Problem variations (Evol-Instruct).
  • bug→fix and fix→bug pairs from real commits.
  • Debug traces: error → investigation → fix.

2.3 Quality filtering

  • License: MITApacheBSD (avoid GPL for closed models).
  • Remove LLM-generated code (self-contamination).
  • Deduplication near-duplicate via MinHash/LSH.
  • Remove secrets, keys, personal data.
  • Filter by the presence of tests and build rate.

2.4 Strategies for data scarcity (scientific domains)

A partnership with a laboratory, hospital, or office is more valuable than any GPU — because it delivers proprietary data.

  1. Active learning — reduces the required dataset by 10–100×.
  2. Simulation as proxy — generate massively; fine-tune on real data.
  3. Foundation models + fine-tuning — do not train from scratch; specialize.
  4. Synthetic data from an LLM — a large LLM generates problems + solutions; filter

    the verifiable ones.

  5. Self-supervised — learn from unlabeled data.

2.5 The principle that runs through everything

Proprietary data is the defensible advantage. Models and compute become commodities; the data specific to your laboratory, hospital, or codebase is the competitive durability. In code, every acceptrejectedit of a suggestion is an "almost-free" RL preference signal — see 06-ai-for-code.kmd.