Experience Replay
Imagine learning to drive from a stream of trips, but being allowed to revisit old lessons rather than learning only from whatever happened five seconds ago. Experience replay gives a reinforcement-learning agent that ability: it stores past interactions and reuses them for training.
How the replay buffer works
After each step, an agent records a transition, commonly written as (state, action, reward, next state, done), in a finite replay buffer. Instead of updating its Q-network only from the newest transition, it repeatedly samples small, random batches from this buffer. In DQN, each sampled transition trains the network toward a Bellman target: the received reward plus the estimated value of the next state. Old transitions are discarded once the buffer reaches capacity, usually in a first-in, first-out pattern.
Why random reuse helps
Consecutive game frames or robot steps are highly related: if an agent drives straight for 100 steps, those 100 examples say nearly the same thing. Training a neural network directly on that sequence can cause updates to chase short-lived patterns and destabilize value estimates. Random replay helps by:
- breaking up these correlations, making batches resemble ordinary supervised-learning data more closely;
- improving sample efficiency, since a rare but informative reward can be learned from many times;
- letting current learning benefit from behaviour produced by earlier, more exploratory policies.
Important trade-offs
A uniform buffer treats every transition equally, even though a rare near-goal mistake can teach more than thousands of routine moves. Prioritized experience replay samples transitions with large prediction errors more frequently, while correcting the sampling bias. Replay also makes learning off-policy: the network learns from data collected under older behaviour, not just its current action choices. That is useful, but stale data can become harmful when an environment changes or the buffer is too dominated by obsolete experience. Combined with DQN’s target network, replay turns a fragile stream of self-generated experience into a reusable training dataset.
Experience replay is a training method that stores an agent’s past transitions—state, action, reward, next state, and terminal flag—in a replay buffer, then samples them randomly for repeated learning updates. By reducing correlations between consecutive experiences and reusing scarce data, it makes neural value-function learning substantially more stable and sample-efficient. It is a core component of DQN and related value-based deep RL methods.
Imagine learning a video game, but after every round you could replay a random selection of past moments: the jump that saved you, the move that lost a life, and the surprise shortcut you found. You would learn from those old experiences again instead of letting them vanish.
Experience replay gives an AI learner a similar kind of memory. It stores past attempts—what it did and what happened next—and revisits a mixed selection later. This helps it learn from important lessons more than once, rather than being overly swayed by only the most recent event. The result is steadier learning and better use of the experience it has already earned.