Double Q-Learning

When an agent chooses the action with the largest predicted value, it can be fooled by its own optimism. Double Q-Learning is a simple fix: keep two independent value estimates so that the estimate choosing an action is not the same estimate judging how good that choice is.

How it reduces overestimation

Ordinary Q-learning updates a value using the largest estimated next-state value:

Q(s, a) ← Q(s, a) + α[r + γ maxₐ′ Q(s′, a′) − Q(s, a)]

The max operator preferentially selects actions whose estimates contain lucky positive noise. Even when every estimate is unbiased on average, the largest one is biased upward. This is called maximisation bias; it can make an agent chase actions that look exceptional only because they were sampled noisily.

Double Q-Learning maintains Q_A and Q_B. On a given update, it updates one table and uses the other to evaluate the selected next action:

Select the next action with Q_A: argmax_a′ Q_A(s′, a′).
Evaluate that action with Q_B.
On another update, reverse the roles of the two tables.

Why this matters in control

Separating selection from evaluation breaks the feedback loop in which one lucky estimate both wins the comparison and supplies its own inflated target. The two tables are still learned from the same experience, so they are not perfectly independent, but the split sharply reduces the systematic optimism of standard Q-learning. During behaviour, the agent can act ε-greedily using the sum or average of both tables.

This matters when rewards are noisy, transitions vary, or many actions have similar true value. In a navigation task, standard Q-learning can repeatedly favour a route that got a few unusually good outcomes; Double Q-Learning produces steadier values and more reliable choices. Its neural-network descendant, Double DQN, uses the online network to select an action and a target network to evaluate it—an important stabilisation in deep RL.

Double Q-Learning is a value-based control algorithm that maintains two independent action-value estimates and uses one to choose the best next action while the other evaluates it. This decoupling reduces the overestimation bias caused by Q-learning’s maximization over noisy value estimates. It matters because more accurate value targets support more stable policy learning and better action selection from reward feedback.

Imagine choosing a restaurant by looking only at its single best review. A lucky, overly enthusiastic review might make an average place seem amazing. Double Q-Learning avoids this kind of mistake when an AI is learning by trial and reward.

Instead of trusting one running opinion about which choice is best, it keeps two separate opinions. One helps pick the promising option, while the other checks how good that option really seems. This makes the learner less likely to be fooled by a choice that got lucky once. The result is steadier learning and fewer overconfident decisions.