Part IV · Ch. 28 — Continuous RL / Control

draft

Reinforcement + connectionist · Robotics/continuous control · Actor-critic. Learns policies in continuous action spaces (torques, velocities). Card: ../02-types-of-ai/05-reinforcement.kmd.

🎨 Figure F-IV.28.0 — Actor and critic. Brief: two modules: the "actor" proposes a continuous action, the "critic" evaluates; a robot/leg learning to walk in the background.

Continuous RL — actor-critic

1. Definition and short history

Extends RL to continuous actions via actor-critic and policy gradient methods. Enabled learned locomotion and manipulation (2017+). The basis of control in robotics (see ch. 34).

2. Foundations

  • Control theory — dynamical systems, stability.
  • Optimization — policy gradient, trust region.
  • Stochastic processes / MDP — stochastic policies, entropy.
  • Physics — dynamics of the controlled body.

3. Algorithms and architectures

  • Policy gradient — optimizes the policy directly.
  • Actor-critic — actor (policy) + critic (value) reduce variance.
  • Algorithms — PPO (stable, standard), SAC (maximum entropy), DDPG/TD3.
  • Stability — trust region (PPO/TRPO), target networks.

4. Inputs

  • Hardware: GPU + massive parallel simulation.
  • Data: simulation rollouts; domain randomization for sim-to-real.
  • Data structures: trajectories, advantage estimates.
  • Systems: physics simulators (MuJoCo, Isaac), vectorized envs.

5. Specialized life cycle

Stage Specialization
0 Problem Continuous control (locomotion, manipulation), safety
1 Data Simulation (cheap) + real data (expensive); randomization
2 EDA State coverage, reward shaping
3 Modeling PPO/SAC; actor-critic architecture; reward
4 Training Parallel rollouts; sensitive hyperparameter tuning
5 Evaluation Reward, robustness, sim-to-real transfer
5.5 Acceptance Physical safety tests, actuation limits
6 Production Real-time policy on the robot; supervision
7 Monitoring Performance, anomalies, wear
8 Retraining New environments, fine-tune on the real one
9 Governance Physical safety, accountability (see ch. 34)

6. Capabilities, modes, and modalities

Bodily-kinesthetic: locomotion, manipulation, industrial control, continuous optimization. The learning engine behind embodied robotics (ch. 34).

7. Limits, risks, and ethics

Sample-inefficient; sim-to-real gap; training instability; reward hacking; safety during exploration in the real world.

8. State of the art and examples

PPO (the "workhorse"), SAC; quadruped/humanoid locomotion, robotic hands; PPO is also the algorithm behind RLHF (ch. 29) — the bridge between control and LLM alignment.