SARSA

SARSA is a way for an agent to learn from the consequences of the choices it actually makes, including its exploratory mistakes. Rather than learning an idealised “always choose the best-looking action” plan, it learns the value of behaving according to its current, imperfect policy.

How the update works
The name SARSA comes from the five pieces of one experience: current State (S), chosen Action (A), received Reward (R), next state (S′), and next action actually selected (A′). It stores an action-value table, Q(s, a), then updates the value of the action just taken:

Q(s, a) ← Q(s, a) + α [r + γ Q(s′, a′) − Q(s, a)]

Here, α controls how strongly new experience changes an estimate, and γ determines how much future reward matters. The crucial detail is Q(s′, a′): SARSA uses the next action drawn by its current behaviour policy, such as an epsilon-greedy policy that usually chooses the best-known action but occasionally explores.

Why “on-policy” matters
SARSA is called on-policy because the policy used to collect experience is the same policy whose values it learns. This makes exploration part of what is being evaluated, rather than treating it as an inconvenience to ignore. In the classic cliff-walking task, a Q-learning agent can favour a narrow route beside a cliff because it assumes future actions will be greedy. A SARSA agent learns that its own exploratory slips could send it over the edge, so it prefers a longer, safer route.

What it gives an agent

• It learns directly from interaction without needing a model of the environment’s dynamics.
• It updates after every step, rather than waiting for an episode to end.
• Its learned behaviour reflects the real cost of continued exploration—important when bad actions are risky.

For small discrete environments, such as a Gymnasium grid world, SARSA is a clear demonstration that an agent must learn not only which actions appear valuable, but also what its own decision-making habits actually lead to.