Notes
🔍

Deadly Triad

Imagine trying to improve a map while using the map itself as the main source of truth. Small errors can feed back into later updates, grow larger, and eventually make the map unreliable. The deadly triad names the particular combination of choices in reinforcement learning that creates this risk.

The three interacting ingredients
The triad consists of:

  • Function approximation: representing a value function such as Q(s, a) with shared parameters, for example a neural network, rather than storing a separate number for every state-action pair.
  • Bootstrapping: updating an estimate using another current estimate, as in a Q-learning target: reward plus a discounted predicted value of the next state.
  • Off-policy learning: learning values for one policy while data comes from another policy, such as learning the greedy policy from exploratory behaviour stored in a replay buffer.

Why this can become unstable
Each ingredient is useful alone. Together, they remove the usual guarantees that value estimates settle down. A neural network generalises an update made at one state to many others; bootstrapping treats a moving prediction as part of its training target; and off-policy data can repeatedly train the network on state distributions unlike those induced by the policy being evaluated. An overestimated value can therefore reinforce itself. In the worst case, values oscillate or explode instead of converging. A classic warning is Baird’s counterexample, where even linear approximation diverges under an off-policy bootstrapped update.

What it means in practice
DQN uses all three ingredients, but practical design choices restrain the feedback loop:

  • a slowly updated target network makes bootstrap targets less volatile;
  • experience replay reduces correlations in consecutive data;
  • conservative learning rates, reward scaling, and clipped losses limit destructive updates.

These measures improve stability; they do not erase the underlying issue. The deadly triad explains why an agent can appear to learn, then abruptly collapse, even while its reported training loss looks well behaved.

The deadly triad is the combination of function approximation, bootstrapping—updating estimates from other learned estimates—and off-policy learning, where data come from a different behaviour policy than the one evaluated. Together, these can make value estimates diverge or become unstable rather than converge. It matters because scalable reinforcement-learning systems commonly use all three, requiring stabilisation methods such as target networks, replay buffers, or constrained updates.

Imagine trying to learn a game from your own past attempts, while also guessing that similar-looking situations should have similar answers. Usually that is useful. But three sensible ideas can combine badly: learning from old or indirect experience, changing your estimates step by step, and using a simplified model to make broad guesses.

This combination is called the Deadly Triad. Together, these choices can make an AI’s predictions drift wildly or become unreliable instead of improving. It matters because an AI may appear to learn at first, then suddenly “forget” good behaviour or make increasingly poor decisions. Researchers design training methods carefully to avoid this unstable mix.