Advantage Function

Imagine choosing between two actions while knowing the situation is generally good. The useful question is not merely “Did I get reward?” but “Was this action better or worse than what I should have expected here?” The advantage function gives that comparison.

For a state s and action a, the advantage is written A^π(s, a):

A^π(s, a) = Q^π(s, a) − V^π(s).

• Q^π(s, a) is the expected future return after taking action a in s, then following policy π.
• V^π(s) is the expected return from that state under the policy’s usual action choices.

A positive advantage says the chosen action beat the state’s normal expectation; a negative one says it underperformed. An advantage near zero says it was about as good as expected. This relative signal is far more informative than raw reward: receiving 10 points is excellent in a state where 2 is expected, but disappointing where 20 is routine.

In REINFORCE and actor–critic methods, the policy is updated to make actions with positive advantage more likely and negative-advantage actions less likely. Subtracting V(s) acts as a baseline: it does not bias the true policy-gradient direction, but it greatly reduces the randomness of gradient estimates. Without it, a policy can overreact to returns caused mostly by luck, delayed events, or an unusually favorable starting state.

The true values are unknown, so an agent learns a value network and computes estimates such as G_t − V(s_t), where G_t is the observed return. PPO commonly uses generalized advantage estimation (GAE), which balances noisy short-horizon temporal-difference errors against lower-variance but more biased estimates. A poor value estimate produces poor advantages: the policy then reinforces actions for the wrong reasons, which can destabilize training even when reward is increasing.