Synthetic Data for Training

Why Synthetic Data?

• Cost: High-quality real data is scarce and expensive to annotate
• Control: Distribution, difficulty, format exactly as desired
• Privacy: No personal data, no leakage risk
• Scalability: Generate millions of examples automatically
• Specific domains: Code, math, reasoning — little high-quality real data

Main risk: If the generator model has a bias, the trained model inherits that bias.

---

Self-Instruct

• arXiv: 2212.10560 | ACL 2023
• Authors: Wang et al. (Allen AI)
• Mechanism: LLM generates instructions → generates instances → filters → fine-tunes itself
• Result: GPT-3 with Self-Instruct approaches InstructGPT at a fraction of the cost
• Impact: Foundation for Alpaca, WizardLM, and every subsequent generation of synthetic data

---

Alpaca

• Paper: "Stanford Alpaca: An Instruction-Following LLaMA Model"
• Authors: Taori et al. (Stanford CRFM) — 2023
• Process: Self-Instruct with GPT-3.5-turbo → 52K instructions · $500 cost
• Model: LLaMA 7B fine-tuned
• Impact: Demonstrated that cheap fine-tuning works; non-commercial license

---

WizardLM — Evol-Instruct

• arXiv: 2304.12244
• Mechanism: Takes simple instructions and "evolves" them (deeper, more specific, more constrained) using an LLM as a mutation operator
• Result: Instructions far more complex than plain Self-Instruct
• WizardCoder: Application to code (Python, C++, Java)
• WizardMath: Application to math

---

Orca — Process Supervision with GPT-4

• arXiv: 2306.02707 (Orca 1) · 2311.11045 (Orca 2) — Microsoft 2023
• Innovation: Explains reasoning step by step (specialized system prompt)
• Data: ~1M examples of full chain-of-thought with GPT-4
• Result: Orca 13B surpasses Vicuna 13B and LLaMA-65B on many tasks

Orca-Math

• arXiv: 2402.14830
• Data: 200K math problems generated via Agent-Instruct
• Mechanism: Specialized agents generate and verify problems
• Result: Phi-2 (2.7B) with Orca-Math surpasses GPT-3.5-turbo on GSM8K

---

Microsoft Phi Series — "Textbooks Are All You Need"

• Phi-1: arXiv:2306.11644 (2023)
  • 7B tokens of "textbook"-style synthetic code generated by GPT-4
  • 1.3B parameters surpass models trained on >100B tokens of code
  • Key finding: The "educational" quality of the data matters more than volume

• Phi-1.5: arXiv:2309.05463
  • 30B synthetic tokens + 20B filtered web
  • 1.3B parameters; common-sense reasoning and math

• Phi-2: 2.7B parameters; synthetic code data + curated NLP
  • Surpasses Mistral 7B on many benchmarks

• Phi-3-Mini: arXiv:2404.14219
  • "Phi-3 Cookbook": synthetic data of didactic quality
  • 3.8B parameters, 128K context; performance of a 7B model

• Phi-4: arXiv:2412.08905 (2024)
  • Multi-stage data synthesis: curation → synthesis → synthesis of synthesis
  • 14B parameters; better at STEM than models 3× larger

---

OpenHermes — General Synthetic Data

• OpenHermes 2.5: 900K high-quality synthetic instructions
  • Generated with Mistral, LLaMA 2, and other models
  • Rigorous curation; state-of-the-art open-source for instruction

---

Generation Methods for Math

MetaMathQA

• arXiv: 2309.12284
• Mechanism: Reformulation of training problems via GPT-4 (reversal, forward reasoning)
• Data: Augmentation of MATH and GSM8K
• Result: LLaMA-2 70B with MetaMathQA surpasses much larger models

OpenMathInstruct

• 1: 1.8M pairs (Mixtral 8x7B → Llama 3)
• 2: Expanded version with Llama 3
• Source: Multiple solving of MATH and AMC/AIME problems with best-of selection

NuminaMath TIR (Tool-Integrated Reasoning)

• Data: 860K competition math problems
• Approach: Writes Python code → executes → incorporates the result into the reasoning
• Winner: MATH Olympiad (competition math) track of NeurIPS 2024

---

Self-Rewarding Language Models

• arXiv: 2401.10020 (Meta 2024)
• Mechanism: The model evaluates its own responses (LLM-as-Judge over itself) to generate preference data iteratively
• Result: Improves each iteration without additional human feedback
• Limitation: Biases self-reinforce without external ground-truth

---

Constitutional AI Synthesis (Anthropic)

• Mechanism: The model critiques its own responses against principles → generates preference data
• Resulting data: (harmfulresponse, revisedresponse) pairs for DPO/RLHF
• Scale: Avoids the need for massive human annotation for alignment

---

Magpie — High-Quality Instruction Data

• arXiv: 2406.08464
• Mechanism: Makes the model generate its own instructions without a seed
• Advantage: Aligned with the response style of the base model itself

---

Comparative Table of Approaches

---

For Kode

• SFT phase: Generate via GPT-4o or Claude examples of code review, refactoring, debugging
• Format: Instruction = repo context + question; Response = code + explanation in CoT
• Volume: 50K–200K high-quality synthetic examples > 1M low-quality ones (the lesson from Phi)
• Automatic verification: Only include examples where the generated code passes the tests