SARSA(Lambda)

SARSA(λ) is a way for an agent to learn not only from its most recent move, but also from a fading memory of the moves that led there. That matters when a reward arrives late: instead of assigning credit only to the last action, the agent can spread credit backward through a recent sequence of decisions.

How the update works
SARSA stands for the sequence State, Action, Reward, next State, next Action. The agent updates its estimate Q(s, a) using the action it actually chooses next:

δ = r + γ Q(s', a') − Q(s, a)

Here, δ is the temporal-difference error: the gap between what the agent expected and what happened. The λ in SARSA(λ) adds eligibility traces. Each recently visited state–action pair receives a trace, which decays by γλ at each step. When a TD error appears, every pair with a nonzero trace is updated, with stronger updates for more recent choices.

Why “on-policy” matters
SARSA(λ) learns the value of the policy the agent is currently following, including its exploratory behavior. If the agent uses ε-greedy action selection and occasionally takes risky random actions, its values reflect that risk. This differs from Q-learning, which updates toward the best-looking next action even if the behavior policy is exploring. In the classic cliff-walking task, SARSA commonly learns a safer path away from the cliff because it accounts for accidental exploratory steps into danger.

Choosing λ in practice
The trace parameter controls how far reward information travels backward:

λ = 0 reduces the method to one-step SARSA: quick, local updates.
λ near 1 gives broader credit assignment, resembling multi-step or full-episode learning.
Intermediate values balance fast learning from recent experience with the stability of shorter backups.

In a small tabular environment such as Gymnasium’s CliffWalking, SARSA(λ) can learn a safe control policy without knowing the transition rules in advance. Its key contribution is practical credit assignment: a good or bad outcome can revise an entire recent chain of decisions, while still learning from the agent’s real exploratory behavior.

SARSA(λ) is an on-policy temporal-difference control algorithm that combines SARSA updates with eligibility traces. Each reward updates not only the most recent state–action pair but also recently visited pairs, weighted by how recently and frequently they occurred. The parameter λ controls how far credit propagates backward. This speeds learning from delayed rewards while learning the value of the policy actually used to explore.

Imagine learning to ride a bike. When you wobble or fall, you do not only learn from the very last movement—you also rethink several moments leading up to it: the turn, the speed, and where you looked. SARSA(Lambda) gives an AI learner a similar kind of memory.

As it tries actions and receives good or bad results, it gives extra credit or blame to recent choices, with the freshest choices counted most. This helps it learn from consequences more quickly, especially when rewards arrive after several steps. It also learns based on the choices it actually makes, including its imperfect exploratory ones, making its behaviour more cautious and realistic.