On-Policy Learning

Imagine learning a new route through a city by following your current best navigation rule, then revising that rule from the trip you just took. On-policy learning works this way: it learns from experience generated by the same decision-making policy it is trying to improve.

The central idea
An RL agent has a policy: a rule mapping a state to an action or action distribution. In on-policy learning, the policy used to collect transitions—states, actions, rewards, and next states—is also the policy whose performance is being evaluated or improved. Because learning changes the policy, this means the agent repeatedly:

• acts using its current policy, including its current exploration choices;
• collects a fresh batch of experience;
• updates that policy from the batch; and
• collects new experience under the revised policy.

What this looks like in practice
SARSA is a classic on-policy value-learning algorithm. Its update uses the next action that the agent actually selects under its current exploratory policy. Thus, if an agent sometimes takes risky exploratory moves, SARSA learns the consequences of that risk. PPO, widely used through libraries such as Stable Baselines3, is an on-policy policy-gradient method: it gathers rollouts from a current policy snapshot, improves that policy for a limited number of passes, then needs fresh rollouts. PPO’s clipping rule limits how far the updated policy can move from the one that produced the data.

Why the constraint matters
On-policy learning gives a clean connection between what the agent practices and what it learns to do. It is especially natural when exploration itself is part of the intended behavior. The cost is data efficiency: old trajectories become stale after substantial policy changes and generally cannot be reused freely. Training a robot in simulation can therefore be practical, while repeatedly collecting fresh real-world trials can be slow or unsafe. Reusing data from a different behavior policy without the required corrections biases learning and can destabilize value estimates or policy updates.