Generalized Advantage Estimation (GAE)

When an agent receives a reward, it needs to decide which recent choices deserve credit. Generalized Advantage Estimation (GAE) is a practical way to produce that credit signal: informative enough to improve a policy, but not so noisy that training lurches in the wrong direction.

What GAE estimates
Actor-critic methods learn both a policy, the actor, and a predicted future return, the critic. To update the actor, they use an advantage: how much better an action turned out than the critic expected from that state. A basic one-step estimate is the temporal-difference residual: δt = rt + γV(st+1) - V(st). GAE combines this residual with later residuals: AGAEt = δt + (γλ)δt+1 + (γλ)2δt+2 + .... The parameter λ controls how far forward that credit assignment reaches.

The bias–variance trade-off
With λ = 0, GAE uses only the immediate residual. Updates have low variance but lean heavily on a potentially inaccurate critic, creating bias. With λ = 1, it resembles a full discounted return: less biased by critic errors, but much noisier because every later reward affects the estimate. Values around 0.95 are common in PPO because they balance these pressures. Put simply, GAE lets the algorithm decide whether to trust its critic now or wait for more evidence from the trajectory.

Why it matters in practice
PPO and A2C commonly compute GAE from short batches of environment interaction, then use the resulting advantages to raise the probability of actions with positive advantage and lower it for negative ones. Without this smoothing, a lucky reward can produce a huge update; with an overly biased setting, the policy can reinforce actions merely because its critic was wrong. GAE does not fix a bad reward function or a critic that cannot learn, but it makes policy-gradient training substantially more stable and data-efficient.