Part IV · Ch. 27 — Policy RL / AlphaZero

draft

Reinforcement + connectionist · Board games/planning · Policy + MCTS. Combines a policy/value network with tree search and self-play for superhuman mastery of games. Card: ../02-types-of-ai/05-reinforcement.kmd.

🎨 Figure F-IV.27.0 — Thinking ahead. Brief: MCTS search tree expanding moves, guided by a network that evaluates positions; a Go board with "move 37" alongside.

AlphaZero — network + search (MCTS)

1. Definition and short history

Unites a neural network (policy + value) with Monte Carlo Tree Search and training by self-play. Milestones: AlphaGo (2016, beats Lee Sedol) → AlphaZero (general) → MuZero (learns the game's model). See Part III, era 5.

2. Foundations

  • Game theory — zero-sum games, perfect information, minimax.
  • Search / MCTS — exploration-exploitation (UCT).
  • Dynamic programming / MDP — value and policy.
  • Probability — self-play generates the training distribution.

3. Algorithms and architectures

  • Dual network — policy (move probabilities) + value (who wins).
  • MCTS — search guided by the network; balances exploring and deepening.
  • Self-play — the agent plays against itself, generating data.
  • MuZero — learns a latent model of the dynamics (without given rules).

4. Inputs

  • Hardware: lots of GPU/TPU for massive self-play.
  • Data: generated by self-play (unlimited in principle).
  • Data structures: search tree, game buffer.
  • Systems: distributed game generation + central training.

5. Specialized life cycle

Stage Specialization
0 Problem Game/planning with clear rules and terminal reward
1 Data Self-play (generates its own data)
2 EDA State coverage, game diversity
3 Modeling Policy+value network, MCTS depth, MuZero?
4 Training Self-play ↔ training loop; lots of compute
5 Evaluation Elo/win rate vs previous versions and humans
5.5 Acceptance Robustness to adversarial strategies, exploitation of flaws
6 Production Policy + MCTS at inference (plans before playing)
7 Monitoring Performance vs new opponents
8 Retraining More self-play; new domains
9 Governance Dual use (strategic planning)

6. Capabilities, modes, and modalities

Strategic/intellectual: superhuman mastery in Go, chess, shogi; planning; optimization (AlphaTensor, algorithm discovery). Combines intuition (network) with deliberation (search) — analogous to System 1/2.

7. Limits, risks, and ethics

Requires a simulable environment and a well-defined reward; enormous compute cost; restricted to domains with clear rules. Generalization outside the game is limited.

8. State of the art and examples

AlphaZeroMuZero; AlphaTensorAlphaDev (algorithm discovery); the "network + search" combination inspires today's reasoning models (deliberate before answering).