TD Error

A TD error is the learning signal that tells a reinforcement-learning agent, “your previous expectation was too high or too low.” It compares what the agent predicted a situation was worth with what actually happened next, including both the immediate reward and a fresh estimate of the future.

How the error is computed
For a state-value estimate, the one-step TD error is:

δₜ = rₜ₊₁ + γV(sₜ₊₁) − V(sₜ)

Here, V(s_t) is the old prediction for the current state, r_t+1 is the reward just received, and γV(s_t+1) is the discounted prediction of what comes next. A positive error means the outcome was better than expected; a negative one means it was worse. TD learning adjusts the value in the direction of this error:

V(sₜ) ← V(sₜ) + αδₜ

The key idea is bootstrapping: rather than waiting until an episode ends to discover the full return, the agent learns partly from its own current estimate of the next state.

Why it drives learning
In a gridworld, an agent may receive no reward until reaching a goal. When it finally gets a reward, TD errors revise the value of the state just before the goal. That changed estimate then creates errors in earlier states, allowing useful information to propagate backward through experience. In action-value methods such as Q-learning and DQN, the same idea becomes:

δₜ = rₜ₊₁ + γ maxₐ Q(sₜ₊₁, a) − Q(sₜ, aₜ)

Practical consequences
TD error is not merely a diagnostic number: it determines the direction and size of each update. Large persistent errors can reveal inaccurate values, changing environment dynamics, or an unstable training setup. In DQN, replay systems can use prioritized experience replay to revisit transitions with large TD errors because they offer the strongest immediate correction. But bootstrapping also creates a risk: an incorrect next-state estimate becomes part of the target, so errors can reinforce one another when learning rates, target updates, or exploration are poorly configured.