Temporal Difference (TD) Learning

Temporal Difference learning is a way to learn from the middle of an experience rather than waiting for the whole story to end. After each step, an agent asks: “Did what just happened make this situation look better or worse than I expected?”

How the update works

TD learning estimates a state’s value: the expected discounted future reward from that state under a policy. After moving from state s to s′, receiving reward r, TD(0) adjusts its estimate using:

TD error:  δ = r + γV(s′) − V(s)
Update:    V(s) ← V(s) + αδ

The quantity δ, the TD error, is the gap between the old prediction and a one-step-improved prediction. The estimate r + γV(s′) uses the current estimate of the next state, so TD is called bootstrapping. Unlike Monte Carlo learning, it does not wait for an episode’s final outcome; unlike dynamic programming, it needs no model of the environment’s transition rules.

Why this matters in reward-driven learning

Rewards can arrive late: a robot gets a useful signal only after completing a route, or an agent in a Gymnasium task scores only at the end. TD lets useful information propagate backward one transition at a time. A state just before reward improves first; then earlier states learn that leading there is valuable. In a finite tabular setting, TD prediction converges under appropriate exploration and learning-rate conditions.

TD(0) learns from one subsequent step.
n-step TD waits several steps, trading faster feedback against more complete returns.
TD(λ) uses eligibility traces to spread credit across recently visited states.

Where it appears—and where care is needed

TD’s core idea drives practical control algorithms: Q-learning updates action values from a greedy next-state estimate, while SARSA updates from the action actually selected next. DQN extends this pattern with neural networks and replay buffers. Bootstrapping also creates a risk: when combined with function approximation and learning from data generated by another policy, value estimates can chase one another and diverge. This is why target networks, careful step sizes, and stable data collection matter in deep RL.

Temporal Difference (TD) learning is a reinforcement-learning method that updates an estimate of a state’s value after each transition, using the observed reward plus the estimated value of the next state. Its TD error measures the gap between this one-step prediction and the current estimate. TD learning enables online, incremental value learning without waiting for an episode to end, forming the basis of methods such as Q-learning and SARSA.

Imagine learning a new route home. You do not wait until you reach your front door to decide whether every turn was good. At each intersection, you use what you have experienced so far and your current sense of whether the route is improving.

Temporal Difference (TD) Learning lets an AI learn in that same “adjust as you go” way. It compares what it expected to happen next with what actually seems likely now, then makes a small correction. This matters when rewards arrive late: a game-playing AI can improve its judgment during a match, rather than waiting until the final win or loss.