Baseline (Policy Gradient)

Policy-gradient learning gets a noisy signal: one successful episode does not prove that every action taken in it was equally good. A baseline gives the algorithm a sensible point of comparison, so it reinforces actions for doing better than expected rather than merely for being present when reward arrived.

How it changes the update
In the basic REINFORCE update, an action is weighted by its later return G: ∇θ log πθ(a|s) · G. With a baseline b(s), it becomes: ∇θ log πθ(a|s) · [G − b(s)]. The bracketed quantity says whether the outcome beat the baseline. When the baseline is the estimated value V(s), this difference is called an advantage estimate: A(s,a) = G − V(s). A positive advantage makes the selected action more likely; a negative one makes it less likely.

Why this is safe—and valuable
Subtracting a baseline does not change the expected policy gradient, provided the baseline does not depend on the particular action sampled. It therefore preserves the learning objective while sharply reducing variance: random luck in returns no longer causes wildly different updates.

A constant baseline, such as the batch’s mean return, is simple but crude.
A state-dependent learned baseline V(s) compares an outcome with what was realistic from that situation.
In actor–critic methods, the critic learns this value estimate while the actor updates the policy.

A well-trained baseline is especially important when rewards are delayed, such as a robot receiving one reward only after completing a task.

Use in modern algorithms
PPO commonly trains a value network and uses advantage estimates, frequently through Generalized Advantage Estimation (GAE). If the baseline is inaccurate, updates become noisier; if its loss is weighted too strongly or shared carelessly with the policy network, it can distort training. But without a baseline, policy gradients can be so high-variance that an agent repeatedly “learns” from chance successes instead of reliably improving its decisions.

A baseline in a policy-gradient estimator is a value subtracted from the sampled return without changing the expected gradient, provided it does not depend on the sampled action. A state-value baseline yields the advantage, measuring how much better an action performed than expected. Baselines matter because they reduce gradient variance, making reward-driven policy updates more stable and sample-efficient.

Imagine coaching a basketball player who takes a shot. Instead of saying only “great!” or “bad!”, you compare the result with what you would normally expect from that shot. Making an easy layup is good, but less surprising than making a difficult shot from far away.

In AI learning, a baseline is that “normal expectation.” It gives the system a reference point for judging whether an action turned out better or worse than usual. This helps it avoid overreacting to random luck or bad breaks. The AI still learns from rewards, but its feedback becomes steadier and clearer—making practice more efficient.