Expected SARSA
Expected SARSA keeps the practical, learn-from-experience spirit of SARSA, but makes its learning target less noisy. Instead of judging a state by one randomly chosen next action, it considers the average outcome of all actions the agent is currently willing to take.
After taking action A in state S, receiving reward R, and arriving at S′, Expected SARSA updates its action-value estimate using:
Q(S,A) ← Q(S,A) + α [R + γ Σₐ π(a|S′) Q(S′,a) − Q(S,A)]The sum is the key difference. For every possible next action a, the algorithm multiplies its current value by the probability that the behavior policy π will select it. With an ε-greedy policy, this includes the small chance of exploratory actions, not just the greedy action.
Plain SARSA uses the value of the one next action actually sampled. That is faithful to the policy but can bounce around because exploration is random. Expected SARSA replaces that sample with its mathematical average, reducing variance. Unlike Q-learning, which uses the value of the best next action regardless of how the agent behaves, Expected SARSA evaluates the same policy that gathers experience. It is therefore an on-policy control method.
In the Cliff Walking benchmark, an ε-greedy agent occasionally makes accidental moves. Expected SARSA learns that a route beside the cliff is risky because its future behavior includes those exploratory slips. Q-learning’s maximum-based target instead imagines perfectly greedy future choices, which can favor the dangerous shortcut during learning.
- Expected SARSA needs the action probabilities under the current policy.
- It is especially convenient in small discrete environments, such as a tabular Gymnasium grid world.
- At a terminal state, there is no future-action expectation: the bootstrap term is zero.
By accounting explicitly for exploration, Expected SARSA learns values that match what the agent will genuinely do, rather than what it would do only if it never made an exploratory choice.
Expected SARSA is an on-policy temporal-difference control algorithm that updates an action-value estimate toward the reward plus the expected value of the next state under the current behavior policy, rather than the value of one sampled next action. This expectation reduces update variance compared with SARSA while still learning values for the policy actually followed. It supports stable policy improvement from reward-driven interaction.
Imagine learning a new board game by playing it. After each move, you do not judge your choice only by the one move you happen to make next. You also consider all the moves you might reasonably make next, and how likely each one is. Expected SARSA brings that same idea to an AI learning through rewards.
It helps the AI estimate how promising a choice is by looking ahead to its usual mix of future choices, rather than betting everything on one lucky or unlucky next move. This often makes learning steadier and less jumpy, especially when the AI is still exploring and sometimes tries imperfect options.