Action-Value Function (Q)

Imagine choosing between two buttons: one gives a small reward now, while the other starts a sequence that pays off much later. The action-value function, written Q, is the learner’s running estimate of how worthwhile each button is in the current situation—not just immediately, but over the future that follows.

What Q measures

For a state s and action a, Q(s, a) is the expected total discounted reward obtained by taking a in s, then continuing according to a policy. “Discounted” means rewards farther ahead are multiplied by powers of a factor γ between 0 and 1, making near-term and distant outcomes comparable. The optimal version, Q*(s, a), assumes the agent acts as well as possible after that first action. Its policy is simple: choose the action with the largest Q-value in the current state.

Learning estimates from experience

An agent does not receive Q-values from the environment; it sees a reward and the next state after acting. It improves an estimate using the Bellman idea: today’s action is worth its immediate reward plus the estimated value of what comes next. In tabular Q-learning, the update moves Q(s, a) toward:

reward received + γ × the largest estimated Q-value at the next state.

This lets a reward at the end of a maze gradually raise the value of earlier actions that led there. DQN uses a neural network to approximate Q-values when states are large, such as image observations in Atari.

Why it matters—and where it can mislead

Q-values turn delayed consequences into a direct action comparison, enabling an agent to decide whether to repeat a known rewarding action or explore an untested one. But they are only estimates. In DQN, each Q-value is updated using another Q estimate, so errors can reinforce themselves; target networks and replay buffers help stabilize this bootstrap process. A poorly designed reward can also produce high Q-values for a shortcut that exploits the scoring rule rather than accomplishing the intended task. Q therefore captures what the agent has learned to pursue, not necessarily what its designer meant.

The action-value function, Q(s, a), is the expected discounted return obtained by taking action a in state s and then following a policy. It evaluates choices rather than states alone, allowing an agent to select actions with the highest predicted long-term reward. Q-values underpin value-based control methods such as Q-learning and define optimal action selection through argmax_a Q(s, a).

Imagine choosing a move in a board game. Before you make it, you naturally wonder: “If I do this now, how likely is it to help me win later?” An action-value function, often called Q, is an AI’s version of that gut feeling.

It gives each possible action a score based on the rewards the learner expects to receive from that choice, including rewards that may arrive much later. For example, a robot might rate turning left as better than turning right because turning left usually leads it closer to its charging station.

By comparing these scores, the learner can choose actions that seem most promising rather than simply reacting to the immediate moment.