Bootstrapping

Imagine learning the value of a route home before reaching the destination. After each turn, you use your current guess about the rest of the journey to improve your guess about the route so far. In reinforcement learning, this is the central idea behind bootstrapping.

Bootstrapping updates a value estimate using another, newer value estimate rather than waiting for the final outcome. In one-step temporal-difference learning, after moving from state s to s′ and receiving reward r, the update is driven by the TD error:

TD error = r + γ V(s′) − V(s)
V(s) ← V(s) + α · TD error

Here, V(s′) stands in for all the future rewards expected after the next state. The learner adjusts its old prediction toward the immediate reward plus this estimated future. The estimate is, in effect, helping to train itself.

Unlike Monte Carlo methods, which wait until an episode ends and use the actual total return, bootstrapping learns after every transition. A robot navigating a long warehouse can improve its estimate after each move instead of waiting until it reaches a charging station. This enables continual learning in tasks with long, unfinished, or even infinite interactions. It is the foundation of TD(0), SARSA, and Q-learning; DQN uses the same basic idea, with a neural network estimating the next-state action values.

Because the target contains an estimate, bootstrapping trades unbiased but delayed feedback for faster, potentially biased learning. Errors can propagate: an inflated value for a next state makes earlier states look better too. With nonlinear function approximation, off-policy data, and bootstrapped targets—the “deadly triad”—updates can become unstable or diverge. DQN addresses part of this problem with a target network, holding the next-state estimator fixed for a while so the learning target does not shift every update. Done well, bootstrapping is what lets an agent turn small pieces of experience into useful predictions long before it has seen every possible outcome.