Target Network

A target network is a deliberately delayed copy of a value network. It gives a learning agent a steadier reference point while the main network is changing, much like measuring progress against a fixed marker instead of one that moves every time you take a step.

In deep Q-learning, the main network estimates how good each action is: Q(s, a). Its update compares its current prediction with a bootstrapped target:

target = reward + discount × max_a' Q_target(next_state, a')

The problem is that, without a target network, the same neural network supplies both sides of this comparison. Every gradient update changes not only the prediction being corrected, but also the number it is trying to match. Errors can reinforce one another, causing Q-values to oscillate, explode, or settle on poor behavior.

A target network holds parameters θ⁻ fixed while the online network, with parameters θ, learns. In DQN, the target parameters are copied from the online network every fixed number of training steps. This makes each batch of temporal-difference updates chase a temporarily stable target.

• Hard updates: copy the online network into the target network every, for example, 10,000 updates.
• Soft updates: slowly blend parameters, using θ⁻ ← τθ + (1 − τ)θ⁻ with a small τ. This is common in actor-critic methods.
• Double DQN: uses the online network to choose the best next action, but the target network to evaluate it, reducing overly optimistic value estimates.

Updating the target network too frequently removes its stabilizing role; updating it too slowly makes its estimates stale and slows learning. Target networks work alongside experience replay: replay reduces correlation among training samples, while the target network prevents the learning objective from shifting too abruptly. Together, these two design choices made it practical for DQN to learn control directly from high-dimensional inputs such as Atari game frames.