Bellman Equation

The Bellman equation is the idea that makes long-term planning feel manageable: instead of evaluating an entire future at once, an agent can evaluate one step now and let its estimate handle the rest. It turns “What will this choice eventually be worth?” into a recursive calculation.

The recursive relationship
For a policy π, the state-value function V^π(s) is the expected total discounted reward from state s onward. The Bellman equation says that value equals:

• the reward expected on the next transition, plus
• the discounted value of the state reached next.

Written compactly: V^π(s) = E[r + γV^π(s′)]. Here, γ is the discount factor: it determines how much future reward counts relative to immediate reward. The expectation covers the policy’s action choices and any randomness in the environment.

Choosing the best future
The Bellman optimality equation replaces “follow policy π” with “choose the best action”: V*(s) = max_a E[r + γV*(s′)]. Its action-value form is especially common: Q*(s,a) = E[r + γ max_a′ Q*(s′,a′)]. This is the target used by DQN: after taking an action, it updates its current Q estimate toward the observed reward plus its own estimate of the best next action.

Why this matters in practice
This one-step “backup” solves the credit-assignment problem incrementally. A robot can learn that reaching a doorway is valuable before it has repeatedly completed the entire route, because value propagates backward from later rewards. But the same mechanism creates a hazard: DQN learns from an estimate that it also produces. With neural networks, changing data, and a moving target, these bootstrapped updates can become unstable or overestimate values. Target networks and replay buffers exist largely to make Bellman-based learning behave reliably.