Policy Gradient Theorem

Imagine adjusting a robot’s behaviour not by telling it the correct move, but by nudging it toward action choices that led to better long-term outcomes. The Policy Gradient Theorem is the result that makes this direct approach practical: it tells us how to improve a policy from reward-bearing experience without needing to differentiate through the environment itself.

What the theorem says
A parameterised policy, written π_θ(a|s), gives the probability of action a in state s. Its goal is to maximise expected return, J(θ). The theorem gives a usable expression for the gradient:

∇θ J(θ) = E[∇θ log πθ(a|s) · Qπ(s, a)]

In plain language: increase the probability of actions in proportion to how valuable they proved to be. The term ∇θ log π_θ(a|s) says how to change the policy’s parameters to favor the sampled action; Q^π(s,a) measures the expected future return after taking it. Crucially, this formula needs samples from the agent’s interaction with the environment, not a differentiable model of its physics or transition rules.

Credit assignment without a teacher
A trajectory can contain hundreds of choices before a reward arrives. REINFORCE uses the observed return as a sample-based substitute for Q^π: rewarding actions that appeared on high-return trajectories and discouraging those on poor ones. In practice, this raw signal is noisy, so implementations use a baseline, commonly a learned value function. Replacing Q with an advantage, A(s,a)=Q(s,a)-V(s), keeps the expected gradient unchanged while reducing variance. This is the actor–critic idea behind methods such as PPO.

Why it matters in practice
The theorem supports stochastic policies and continuous actions, where choosing among a finite table of actions is impossible. A MuJoCo walker, for example, can directly learn how to adjust joint torques. But the theorem does not guarantee calm learning: inaccurate value estimates, huge gradient steps, or rewards that accidentally favor a shortcut can push the policy toward collapse. PPO constrains each update so that reward-driven improvement does not erase useful behaviour in one unstable leap.

The Policy Gradient Theorem expresses the gradient of a parameterized policy’s expected return as an expectation over actions sampled from that policy, weighted by their action values. It enables direct optimization of stochastic policies using trajectory data without differentiating through the environment’s dynamics. This result underpins policy-gradient algorithms such as REINFORCE, actor–critic methods, and modern continuous-control RL.

Imagine learning to throw a basketball: after many shots, you notice that certain small changes—aiming a little higher or using less force—make baskets more likely. You do not need someone to tell you the perfect motion for every throw. You just need feedback on what tends to work.

The Policy Gradient Theorem is a key idea that gives AI a reliable direction for making its choices better. A policy simply means the AI’s habit of choosing actions. The theorem links changes in that habit to changes in the rewards it earns, helping the AI gradually favor decisions that lead to better outcomes.