Post-Training: SFT, Alignment, and Reasoning

Supervised Fine-Tuning (SFT)

• Approach: instruction + response pairs; standard cross-entropy loss
• Data: UltraChat, WizardLM, OpenMathInstruct, The Stack v2 (for code)
• Starting point: Always begins here before any preference optimization

---

Preference Optimization Methods

RLHF — Reinforcement Learning from Human Feedback

• arXiv: 2203.02155 (InstructGPT)
• Pipeline: SFT → Reward Model → PPO
• Problem: Complex, unstable, requires a separate reward model
• When to use: When you need fine-grained control of behavior (e.g., Claude-level safety)

DPO — Direct Preference Optimization

• arXiv: 2305.18290 | NIPS 2023
• Authors: Rafailov et al.
• Mechanism: Reframes RLHF as classification; eliminates the reward model
• Required data: (preferred, rejected) pairs per prompt
• Advantage: Much simpler than PPO; near-equivalent quality
• Used by: Fine-tuning of Llama 3, Qwen 2.5, most modern models

KTO — Kahneman-Tversky Optimization

• Advantage over DPO: Works with binary feedback (good/bad) without pairs; data easier to collect
• Basis: Kahneman-Tversky utility model (prospect theory)

ORPO — Odds Ratio Preference Optimization

• Mechanism: Combines SFT loss + odds ratio penalty in a single stage
• Advantage: Eliminates the separate SFT phase; trains in one step

SimPO — Simple Preference Optimization

• Mechanism: Uses average per-token likelihood as reward; adds a target reward margin
• Advantage: Simpler than DPO; no reference model

IPO — Identity Preference Optimization

• Focus: Data alignment for preference learning

---

Reinforcement Learning with Verifiable Reward (RLVR)

PPO — Proximal Policy Optimization

• arXiv: 1707.06347
• When to use: Strong behavioral shaping; when the reward model is reliable
• Cost: High — requires policy model + value model + reward model in memory

GRPO — Group Relative Policy Optimization

• arXiv: 2402.03300 (DeepSeekMath)
• Mechanism: Eliminates the critic model; estimates the baseline from group scores
• Advantage: More memory-efficient than PPO; especially good for reasoning
• Used by: DeepSeek-R1, DeepSeek-Math, many reasoning models in 2025

REINFORCE++

• arXiv: 2501.03262
• Variant: Stabilizes classic REINFORCE for large LLMs

DAPO (ByteDance)

• arXiv: 2503.14476
• System: Large-scale open-source RL; adaptive clipping, token-level policy gradient
• Problem solved: Instabilities in training reasoning models with long CoTs (reward collapse, entropy collapse)
• Results: 50 points on AIME 2024; surpasses DeepSeek-R1-Zero with 50% fewer training steps
• Key techniques: Dynamic Sampling Policy Optimization (DSPO) + Clip-Higher heuristic + token-level loss for long CoTs
• When to use: RLVR training when CoTs are very long (>2K tokens) and PPO/GRPO collapse

---

Reward Models

Outcome Reward Models (ORMs)

• Feedback: Only on the final result (sparse)
• Problem: Does not localize intermediate errors

Process Reward Models (PRMs)

• arXiv: 2305.20050 (Let's Verify Step by Step)
• Feedback: At each reasoning step (dense)
• Advantage: Localizes errors; better interpretability; enables search at inference
• Result: Smaller models with PRM outperform larger models with ORM

---

Constitutional AI (CAI)

• arXiv: 2212.08073 | Anthropic 2022
• Mechanism: The model critiques and revises its own outputs using predefined principles (a "constitution")
• Benefit: Scalable alignment without constant human labels
• Variant: RLAIF — AI feedback replaces human feedback in the reward model

---

LLM-as-Judge

• Concept: An LLM evaluates the quality of responses from other LLMs
• Surveys: arXiv:2412.05579 (LLMs-as-Judges: Comprehensive Survey)
• Types: Single-LLM, Multi-LLM, Agent-as-a-Judge
• Common biases: Position bias, verbosity bias, self-preference bias
• Usage: AlpacaEval 2.0, Arena-Hard — based on LLM-as-Judge

---

Test-Time Compute (Inference Scaling)

• Concept: Spending compute at inference often yields more than scaling up the model
• arXiv: 2408.03314 (Scaling Test-Time Compute)
• Techniques:
  • Multiple samples + verification (best-of-N)
  • MCTS over text
  • Beam search with PRM
  • Extended thinking (Claude, o1/o3)
• Models based on this: OpenAI o1/o3, Claude Extended Thinking, DeepSeek-R1

---

Reasoning Paradigm (2025–2026)

The biggest leap in reasoning comes from RLVR with verifiable rewards:

1. High-quality reasoning data: long verifiable CoTs (olympiad math, code)
2. Process Reward Models: Feedback at each step
3. RLVR (GRPO/PPO): reward = "passed or did not pass" the tests
4. Test-time search: beam search + formal verifier

Realistic cost to reproduce R1-style: 2× RTX 4090 for 2–4 weeks of experimentation.

---

Pipeline Recommendation for Kode