TD(Lambda)

TD(λ) is a way to help an agent learn from consequences that unfold over more than one step, without waiting until an entire episode ends. It sits between two useful extremes: updating immediately from the next state, and waiting to see the full return.

Plain TD(0) updates a state’s value estimate using one later estimate. Its core signal is the TD error:

δₜ = rₜ₊₁ + γV(sₜ₊₁) − V(sₜ)

TD(λ) blends updates based on 1 step, 2 steps, 3 steps, and longer sequences. The parameter λ, from 0 to 1, controls the blend:

• λ = 0: TD(0), which learns quickly but relies heavily on its current, imperfect value estimates.
• λ close to 1: gives more weight to longer observed outcomes, approaching Monte Carlo learning when λ is 1 for episodic tasks.

This creates a practical bias–variance trade-off: shorter updates bootstrap more and have lower variance; longer updates use more real experience but can be noisier.

The usual online implementation uses eligibility traces. Think of a trace as a fading memory of recently visited states. Each time step, the current state’s trace increases, old traces decay by γλ, and the current TD error updates every state in proportion to its trace. A reward discovered now can therefore adjust not only the latest state, but also the earlier decisions that plausibly led to it.

In a maze, an agent might reach a goal only after twenty moves. TD(0) pushes credit backward one state at a time; Monte Carlo waits for the goal before updating anything. TD(λ) propagates credit backward through the recent path as learning happens. This can greatly speed tabular prediction and control, though large λ values can spread misleading errors when value estimates or behavior change rapidly. Eligibility traces are also the foundation behind variants such as Sarsa(λ), which learn action values while balancing exploration and known rewarding actions.