Part IV · Ch. 26 — Deep RL (value)

draft

Reinforcement + connectionist · Games/control · Neural network + value function (Q). Learns which action has the highest expected value in each state, by trial and error. Card: ../02-types-of-ai/05-reinforcement.kmd.

🎨 Figure F-IV.26.0 — Learning through reward. Brief: agent in an environment (e.g., Atari), receiving state/reward and choosing actions; Q network mapping state→value of each action; replay buffer alongside.

Deep RL (value)

1. Definition and short history

Estimates the action-value function Q(s,a) with a neural network. Milestone: DQN playing Atari directly from pixels (DeepMind, 2013/2015), which kicked off deep RL (see Part III, era 5).

2. Foundations

  • Markov decision processes (MDP) — states, actions, reward, transition.
  • Dynamic programming — the Bellman equation.
  • Probability / stochastic processes — policy, expected return.
  • Psychology — operant conditioning (reward).

3. Algorithms and architectures

  • Q-learning — Bellman update; off-policy.
  • DQN — network approximates Q; experience replay + target network

    stabilize it.

  • Improvements — Double DQN, Dueling, Prioritized Replay, Rainbow.
  • Exploration — ε-greedy, intrinsic motivation.

4. Inputs

  • Hardware: GPU; many simulation steps (environment).
  • Data: generated by interaction (unlabeled); replay buffer.
  • Data structures: replay buffer (ring), state tensors.
  • Systems: simulatorsenvironments (GymALE), rollout parallelization.

5. Specialized life cycle

Stage Specialization
0 Problem Environment, discrete action space, reward function
1 Data Generated by exploration; replay buffer
2 EDA Reward distribution, reward sparsity
3 Modeling Q network architecture, improvements (Rainbow), exploration
4 Training Interaction↔update; unstable; target network
5 Evaluation Average reward, sample efficiency, stability
5.5 Acceptance Robustness to unseen states, safety
6 Production Fixed policy (greedy in Q); lightweight inference
7 Monitoring Performance, environment distribution shift
8 Retraining Environment changed → retrain
9 Governance Safety of the learned behavior

6. Capabilities, modes, and modalities

Reactive control in discrete actions: games, simple navigation, policy optimization; learns from scratch through experience, without labels.

7. Limits, risks, and ethics

Sample-inefficient (many episodes); unstable; sensitive to reward design (reward hacking); hard in continuous actions (→ ch. 28). Safety during exploration.

8. State of the art and examples

DQN/Rainbow (Atari), R2D2, Agent57; the conceptual basis of modern RL; value combines with policy in actor-critic methods (ch. 28).