Advantage Actor-Critic (A2C)

Imagine learning to play several copies of the same game at once, then pausing to combine what all of them discovered. Advantage Actor-Critic (A2C) does this while separating two jobs: choosing actions and judging whether those actions turned out better or worse than expected.

Two networks, one learning signal

The actor is the policy, π(a|s), which assigns probabilities to actions in a state. The critic estimates the expected future reward, V(s). Rather than rewarding an action simply because a trajectory earned reward, A2C uses an advantage: the observed return minus the critic’s estimate. A positive advantage means “this action worked out better than predicted,” so the actor becomes more likely to select it; a negative one reduces its probability.

The critic learns to make its value predictions closer to observed, usually n-step, returns.
The actor is updated with the advantage-weighted policy-gradient signal.
An entropy bonus commonly keeps the policy from becoming prematurely deterministic, preserving exploration.

Why the “A2C” part matters

A2C is the synchronous version of A3C. It runs several environment instances in parallel, collects a short rollout from each, aggregates their gradients, and performs one shared update. This creates more varied training data than a single trajectory and makes efficient use of vectorized simulators. In a Gymnasium or MuJoCo task, one worker might discover that moving left avoids an obstacle while another exposes a failure case; the joint update reflects both. Libraries such as Stable Baselines3 implement A2C with these parallel environments directly.

What it fixes—and what remains difficult

Using advantages reduces the noisy “credit assignment” problem of plain policy gradients: winning after ten actions does not mean every action deserves equal praise. Yet A2C is on-policy: it learns only from data produced by its current policy, then discards that data after updating. Poor critic estimates can therefore mislead the actor, and overly large learning rates or insufficient exploration can cause unstable training or a policy that exploits an accidental reward loophole. A2C remains a clear, practical foundation for understanding more heavily constrained actor-critic methods such as PPO.

Advantage Actor-Critic (A2C) is a synchronous actor-critic algorithm in which a policy network selects actions and a value network estimates expected returns. It updates the policy using the advantage—the observed return minus the critic’s value estimate—so actions are reinforced only when they perform better than expected. A2C reduces gradient variance and improves training stability while parallel environment workers collect experience efficiently.

Imagine learning a new game with two helpers. One says, “Try this move next,” while the other watches and says, “That turned out better—or worse—than we expected.” Advantage Actor-Critic (A2C) gives an AI those two roles: an “actor” that chooses actions and a “critic” that judges the results.

The key idea is the advantage: not just whether a result was good, but whether it was better than the AI had reason to expect. This helps it notice which choices truly deserve more practice, making learning from wins, losses, and surprises more efficient.