Value Function
A value function is an agent’s forecast of how rewarding the future looks from where it is now. Rather than judging an action only by its immediate payoff, it asks: “Starting here, and continuing to behave this way, how much reward should I expect to collect over time?”
What it estimates
For a policy π, the state-value function Vπ(s) is the expected discounted sum of future rewards after arriving in state s and then following that policy. Discounting, usually written with a factor γ between 0 and 1, makes near-term rewards count more and keeps an endless future finite. The closely related action-value function Qπ(s, a) estimates the return from taking action a in state s, then following π.
How learning works
A value function connects delayed consequences to present decisions. It is commonly learned through the Bellman relationship: a state’s value should equal its immediate reward plus the discounted value of the state reached next. This lets an agent update an estimate from another estimate, called bootstrapping. For example, DQN learns Q-values and selects actions with the largest predicted Q-value. Its update effectively says: “This move looked good because it gave reward now and led to a state predicted to be good later.”
- In a maze, a corridor with no immediate reward can gain value because it leads toward the exit.
- In a game, a move that sacrifices points now can be preferred when it creates a winning position later.
- In control tasks such as MuJoCo locomotion, value estimates help PPO decide which trajectories deserve stronger policy updates.
Why it is difficult and important
Rewards alone do not label the “correct” action at each moment; the value function supplies a learned measure of long-term credit. But its targets move as the policy and estimates change. With inaccurate bootstrapped targets, limited exploration, or unstable function approximation, values can drift or become wildly overoptimistic—one reason DQN uses target networks and replay buffers. A well-calibrated value function makes long-horizon planning practical; a poor one can make an agent confidently repeat actions that only appear rewarding.
A value function estimates the expected cumulative discounted reward obtainable from a state, or from a state–action pair, under a policy. State values are written V(s); action values are written Q(s,a). It matters because it predicts long-term consequences beyond immediate reward, allowing an agent to compare choices and improve its policy toward higher-return behaviour.
Imagine choosing a route home. One street may look fine right now, but you remember it usually leads to traffic. Another route may take a little longer at first but reliably gets you home sooner. A value function is like that learned sense of “how promising is this situation?”
For a learning system, it estimates how much future reward it can expect from where it is now—or from taking a particular action. It matters because good decisions are not always the ones with the biggest immediate payoff. A robot may take an extra step now to reach a much better outcome later. The value function helps it look beyond the next moment.