Notes
🔍

TRPO

TRPO is built around a simple safety rule: when improving a policy, do not change it so drastically that yesterday’s useful experience becomes misleading. In deep reinforcement learning, a large gradient step can turn a competent controller into one that suddenly falls, crashes, or stops collecting useful data.

The trust-region idea
Trust Region Policy Optimization (TRPO) improves a stochastic policy while restricting how far the new policy can move from the old one. It maximizes a reward-improvement estimate—the surrogate objective—subject to a limit on the average Kullback–Leibler (KL) divergence between the old and new action distributions. In effect, the algorithm asks: “Which update looks most rewarding, while still behaving recognizably like the policy that generated this batch of experience?”

What an update does

  • Run the current policy in the environment and collect trajectories of states, actions, and rewards.
  • Estimate which actions were better or worse than expected using an advantage estimate.
  • Increase the probability of advantageous actions and decrease that of disadvantageous ones.
  • Constrain the update’s KL divergence, using a natural-gradient-style direction, conjugate-gradient optimization, and a line search to respect the constraint.

Why the constraint matters
Without the constraint, a policy-gradient network can exploit noise in one batch: a robot that happened to take a lucky step might receive an update so large that its walking behavior collapses. TRPO’s conservative update makes learning more stable, especially in continuous-control simulators such as MuJoCo. Its theory provides a bound connecting small policy changes to safer expected-performance improvement, though neural-network approximation and imperfect estimates still prevent a guarantee in every run. TRPO is on-policy: after each major update it needs fresh data from the current policy, which makes it less sample-efficient than replay-buffer methods. PPO became more widely used because it approximates TRPO’s caution with a simpler clipped objective, while retaining the same central lesson: stable learning requires limiting policy drift.

TRPO (Trust Region Policy Optimization) is an on-policy reinforcement-learning algorithm that improves a policy while constraining how far the updated action distribution can move from the previous one, typically using a KL-divergence limit. This trust-region constraint makes policy updates more stable and reduces performance collapse from overly large gradient steps. TRPO established a principled foundation for stable neural policy optimization and motivated simpler methods such as PPO.

TRPO, short for Trust Region Policy Optimization, is like teaching someone to ski without letting them suddenly try a dangerous new technique on a steep slope.

When an AI learns by trial and reward, it needs to change its behaviour as it discovers what works. But changing too much at once can make it forget useful habits and perform far worse. TRPO encourages careful, limited adjustments: keep what has been working, test a slightly improved approach, and avoid wild swings.

This makes learning more dependable, especially in complicated tasks where one reckless update could undo a lot of progress.