TD

TD stands for temporal-difference learning: a way for an agent to improve its predictions while an episode is still unfolding. Rather than waiting until a game ends or a robot finishes its route, it learns from the difference between what it expected a moment ago and what the next moment now suggests.

The central update
For a state-value estimate V(s), the basic TD(0) update is driven by the TD error: δ = r + γV(s') - V(s). Here, r is the reward just received, γ discounts future rewards, and V(s') is the current estimate of the next state’s value. The agent changes its old prediction in the error’s direction: V(s) ← V(s) + αδ. This is called bootstrapping: an estimate is updated using another estimate, rather than a final known outcome.

Why learn this way?
TD sits between two familiar extremes:

Monte Carlo learning waits for a complete episode and uses the actual final return.
Dynamic programming requires a model of the environment’s transition and reward rules.
TD needs neither a completed episode nor a known model; it updates immediately from experience.

In a gridworld, stepping into a square that is better than predicted creates a positive TD error, raising the value of the previous square. A worse-than-expected step creates a negative error. This lets credit move backward through experience, one transition at a time.

Practical importance and limits
TD learning is the engine behind methods such as SARSA and Q-learning; DQN trains its action-value network from a TD error. Its immediate updates make it data-efficient, especially when long episodes make waiting expensive. But bootstrapping also creates a feedback loop: inaccurate estimates train other estimates. With neural networks, changing policies, or correlated replay data, this can become unstable or diverge unless details such as target networks and learning rates are handled carefully. TD therefore makes online reward learning practical, while demanding care about the moving targets it creates.

TD (temporal-difference learning) is a reinforcement-learning method that updates a state’s value estimate after each transition using the difference between its current prediction and a reward-plus-next-state prediction, called the TD error. It learns from incomplete episodes by bootstrapping from existing estimates. TD enables online value learning in continuing tasks and underlies methods such as Q-learning and SARSA.

Imagine learning a new route home. You do not wait until you arrive to decide whether every turn was good or bad. At each intersection, you use what you have learned so far: “This street seems to be taking me in the right direction.”

TD, short for temporal difference, is an AI learning idea built around that same instinct. The system updates its expectations step by step, comparing what it expected a moment ago with what seems likely now. This lets it learn from incomplete experiences rather than waiting for a final result. It is especially useful when rewards come late, such as learning to play a long game or navigate a maze.