Part IV · Ch. 29 — RLHF / RLAIF

draft

*Reinforcement + connectionist · Language alignment · Reward model + PPO/DPO.* Aligns LLMs (ch. 01) by using human (or AI) feedback as reward. Card: ../02-types-of-ai/05-reinforcement.kmd.

🎨 Figure F-IV.29.0 — From feedback to behavior. Brief: humans comparing two responses → reward model learns the preference → LLM is optimized to maximize it; arrows of the RL loop.

RLHF / RLAIF

1. Definition and short history

Turns human preferences into a reward signal to adjust the behavior of an LLM. Milestone: InstructGPT (2022) and ChatGPT; Constitutional AI / RLAIF (Anthropic) uses principles and AI feedback. It is what turns a raw LLM into a helpful, honest, and safe assistant.

2. Foundations

  • Decision/utility theory — preferences → reward function.
  • Optimization / RL — PPO; or DPO (direct optimization, without explicit RL).
  • Psychometrics — quality and consistency of the human label.
  • Ethics / philosophy — which values to align to, and whose.

3. Algorithms and architectures

  • Preference collection — humans (or AI) compare responses.
  • Reward model — learns to predict the preference.
  • Optimization — PPO maximizes reward with a KL penalty to the base model

    (avoids over-optimization).

  • DPO — derives the policy directly from preferences (simpler/more stable).
  • RLAIF / Constitutional AI — AI-generated feedback guided by principles.

4. Inputs

  • Hardware: GPU (training the reward model + optimizing the LLM).
  • Data: preference pairs (human/AI); diverse prompts.
  • Data structures: comparison datasets, reward scores.
  • Systems: annotation infra, RL/DPO pipeline, alignment eval.

5. Specialized life cycle

Stage Specialization
0 Problem Define target behavior (helpfulhonestharmless) and principles
1 Data Collect preferences (human labelers or AI); guidelines
2 EDA Inter-annotator agreement, topic coverage, bias
3 Modeling Reward model; choose PPO vs DPO; constitution (RLAIF)
4 Training Train reward → optimize LLM (with KL to base)
5 Evaluation Human win rate, safety, appropriate refusal, reward hacking
5.5 Acceptance Red team, jailbreak, bias, refusal calibration
6 Production Aligned model served (it's phase 4/post-training of the LLM)
7 Monitoring Refusal rate, complaints, behavior drift
8 Retraining New preferences/policies; new iteration
9 Governance Which values, whose; transparency; sycophancy; safety

6. Capabilities, modes, and modalities

Not an autonomous "type", but rather the alignment stage that shapes the behavior of LLMs and multimodal models (ch. 01, 14). It defines helpfulness, tone, safety, and refusal.

7. Limits, risks, and ethics

Reward hacking and over-optimization; sycophancy (flattery); bias from labelers; "align to which values?" is an open question; annotation cost. RLAIF reduces cost but inherits biases from the judge model.

8. State of the art and examples

InstructGPTChatGPT (RLHF), Claude (Constitutional AIRLAIF), DPO and variants (simplify the pipeline); a central area of AI safety and alignment, in rapid evolution.

Reinforcement Paradigm complete (ch. 26–29). Part IV index: INDEX.kmd.