Policy-Based RL
Rather than first learning how good every possible move is and then choosing the best one, policy-based RL learns the decision rule itself. It is like training a driver’s instincts directly: given what they see, what should they do next?
The central idea
A policy, written as πθ(a|s), maps a state s to an action a, with neural-network parameters θ. In a stochastic policy, it produces a probability distribution over actions; in a continuous-control task, it might output the mean and spread of a Gaussian distribution over motor commands. Training adjusts θ to maximize expected return: the total discounted reward obtained over complete interactions with the environment.
How it learns from reward
Policy-gradient methods raise the probability of actions that led to better-than-expected outcomes and lower it for actions that did poorly. REINFORCE uses complete episode returns directly, while actor–critic methods learn a value estimate—the critic—to judge whether an action was better than expected. This produces an advantage signal, reducing the noisy updates caused by rewards arriving long after the relevant action.
- PPO is a widely used policy-based method that limits how far the policy can change in one update, helping prevent destructive jumps.
- Policy-based methods handle continuous actions naturally, such as steering, torque, or robot-joint positions in MuJoCo.
- Because the policy generates its own training data, exploration is built into stochastic action selection—but poor exploration still leaves it stuck with sparse rewards.
Practical trade-offs
Unlike value-based DQN, which selects actions by maximizing learned action values, policy-based RL can represent rich, smooth, directly optimized behavior. Its price is high-variance, data-hungry learning: a critic that drifts can push the policy in the wrong direction, and a reward loophole can be amplified into bizarre behavior. A robot policy that gains reward for moving forward, for example, might learn to exploit a simulator glitch rather than walk. Careful reward design, stable updates, and evaluation under slightly changed dynamics are therefore essential.
Policy-Based RL directly represents and optimizes a policy: a mapping from states or observations to actions, usually as a parameterized probability distribution. It adjusts policy parameters to maximize expected cumulative reward rather than first learning action values and deriving behavior from them. This enables direct learning of stochastic or continuous-action decisions, but optimization can be noisy and sensitive to unstable reward estimates.
Imagine teaching someone to ride a bike by encouraging the movements that keep them balanced, rather than first giving them a detailed score for every possible body position. Over time, they develop a feel for which actions to take in each moment.
Policy-Based RL is AI learning in that spirit. A policy is simply the learner’s habit or plan for choosing an action in a situation: turn left here, slow down there, reach for this object now. Instead of mainly judging how good each situation is, it focuses directly on improving those choices. This is especially useful when actions need to be smooth, flexible, or naturally varied, such as controlling a robot or playing a fast-moving game.