Value-Based RL

Imagine choosing actions by asking, “From here, which move is likely to lead to the most reward later?” Value-Based RL learns to answer that question directly. Rather than first learning a complete plan or explicitly representing the action-selection rule, it learns scores that rank the long-term promise of possible actions.

What the algorithm learns
The central object is an action-value function, written Q(s, a). It estimates the expected total discounted reward after taking action a in state s, then continuing well afterward. At decision time, the agent selects the action with the highest estimated value:

State: a game screen, robot position, or inventory level.
Action: move left, grasp an object, or reorder stock.
Value: “How rewarding will this choice prove over the rest of the interaction?”

In Q-learning, an observed reward updates the estimate toward reward + discounted value of the best next action. This lets reward information flow backward: a reward found at the end of a maze gradually raises the values of earlier turns that led there.

Learning while acting
The catch is that values are only estimates, and the agent needs experience to improve them. It therefore balances exploration—trying uncertain actions—with exploiting the action currently scored highest. A common rule is epsilon-greedy: usually choose the best-valued action, but occasionally act randomly. Without exploration, an agent can miss a better route forever; without exploitation, it spends too much time on known-bad choices. Because the update uses one learned estimate to update another, called bootstrapping, inaccurate values can reinforce one another when training is unstable.

From tables to deep networks
For small environments, Q-values fit in a table. For images or continuous sensor inputs, Deep Q-Networks (DQN) use a neural network to approximate Q-values. DQN became well known for playing Atari games from pixels, using experience replay and a separate target network to reduce destructive feedback in training. Value-based methods work naturally when actions are discrete, such as choosing among game controls or menu options. They are less direct for precise continuous controls, where maximizing over every possible action is difficult. A practical failure mode is reward hacking: if points reward spinning in place rather than finishing a task, Q-values faithfully learn to prefer spinning—revealing a flawed reward, not a flawed optimizer.

Value-Based RL learns a value function that estimates expected cumulative reward from a state or state–action pair, then selects actions with the highest estimated value. Rather than directly optimizing a policy, it improves decisions through these reward predictions. It matters because accurate value estimates support action selection and policy improvement; errors or instability in them can produce poor, divergent behaviour. Q-learning and Deep Q-Networks are value-based methods.

Imagine learning which route home is best by trying different streets. You do not need someone to tell you every turn to take. Instead, you gradually remember which choices tend to get you home faster, avoid traffic, or feel safer.

Value-Based RL works in that spirit. It helps an AI estimate the likely long-term payoff of each possible choice: “How good is this move likely to be, considering what may happen next?” The AI then usually picks the option with the highest estimated value. This matters when rewards arrive later, such as winning a game after many moves, rather than immediately after one decision.