Actor-Critic
Think of learning to play a game with two teammates: one chooses the next move, while the other judges how promising the situation is. Actor-Critic methods split these jobs so an agent can improve its behaviour without waiting until the very end of an episode to learn whether every decision was good.
Two networks, one learning loop
The actor is the policy, written as πθ(a|s): given state s, it selects an action a. The critic estimates future reward, commonly through a state-value function Vπ(s). After acting, the agent observes reward r and a new state s'. The critic computes a prediction error:
δ = r + γV(s') - V(s).
This temporal-difference error says whether events went better or worse than the critic expected. The critic adjusts its value estimate toward the observed outcome; the actor raises the probability of actions with positive errors and lowers it for negative ones.
Why the pairing helps
A pure policy-gradient method can use a complete episode return, but that signal is noisy: a good action can be followed by bad luck. The critic supplies a learned baseline, yielding an advantage: how much better an action was than the state’s usual prospects. This makes updates more targeted and less variable. In contrast, value-based methods such as DQN choose actions indirectly through estimated action values; an actor directly represents a stochastic or continuous-action policy, which is especially useful for MuJoCo control tasks.
Practical trade-off
Actor-Critic is the foundation of widely used algorithms such as A2C, A3C, and PPO. PPO constrains how far the actor changes per update, while techniques such as generalized advantage estimation balance noisy short-term errors against delayed reward. The cost is a moving-target problem: the actor learns from the critic’s judgment, but the critic is itself imperfect and changing. A biased or unstable critic can push the policy toward a reward loophole or cause training collapse, which is why reward design, normalization, and conservative updates matter so much.
Actor-critic is a reinforcement-learning architecture with two learned components: an actor that selects actions through a policy, and a critic that estimates action or state value to evaluate them. The critic supplies a lower-variance learning signal, commonly an advantage estimate, for updating the actor. This enables efficient policy learning in large or continuous action spaces while reducing the instability of pure policy-gradient updates.
Imagine learning to play a new video game with two helpful voices. One voice, the actor, chooses what to do next: jump, turn, or pick up an item. The other, the critic, watches the result and says, “That was better than expected,” or “That move did not pay off.”
An Actor-Critic system gives an AI both roles. The actor tries actions, while the critic helps judge whether those choices seem to be leading somewhere good. This is useful because rewards can be delayed: you may only win points long after making an important earlier move. Together, the two roles help the AI improve its decisions through practice and feedback.