Bellman Optimality Equation

Imagine pausing a game at a particular situation and asking: “What is the best total reward I can still earn from here?” The Bellman Optimality Equation answers that question by breaking a long-term decision into one immediate choice plus the best possible future that follows.

The core idea
For an optimal state-value function, the equation is:

V*(s) = max_a [ R(s, a) + γ Σ_s′ P(s′ | s, a)V*(s′) ]

Here, V*(s) is the greatest expected discounted return achievable from state s. For every available action a, the agent considers:

• the immediate reward, R(s, a);
• each possible next state s′, weighted by its transition probability P(s′ | s, a);
• the value of continuing optimally afterward, V*(s′);
• the discount γ, which determines how much future reward counts now.

The max is what makes this an optimality equation: it assumes the agent chooses the best action at every future decision point, not merely the action used by its current policy.

Why it drives RL algorithms
The equation supplies the target that planning and value-learning methods try to satisfy. In a gridworld, an agent learns that stepping toward a goal can be worthwhile even when each step costs reward, because the next state has higher long-term value. In Q-learning and DQN, the action-value version produces the familiar update target: immediate reward plus γ max_a′ Q(s′, a′). That bootstrap—updating an estimate from another estimate—lets an agent learn from partial experience rather than waiting for every episode to finish.

The important catch
The equation is exact when the environment’s rewards and transition probabilities are known. In real learning, they must be estimated from interaction, while the agent’s choices determine which states it observes. Sparse exploration can leave the “best” action untried; noisy neural value estimates can amplify errors through the max operation, producing overestimation or instability. Techniques such as target networks in DQN exist largely to make this Bellman-based learning process behave reliably.