Prioritized Experience Replay

When an agent learns from a replay buffer, not every past experience is equally useful. Prioritized Experience Replay gives extra attention to transitions that the agent currently finds surprising or difficult, rather than sampling its old experiences uniformly at random.

How it works

In a DQN-style agent, the replay buffer stores transitions such as (state, action, reward, next state). Standard experience replay draws these uniformly, breaking up correlations in consecutive gameplay and allowing old data to be reused. Prioritized replay instead assigns each transition a priority, usually based on its temporal-difference (TD) error: the gap between the Q-value currently predicted and the target implied by the observed reward and next state. A large error means “this experience taught me something my value estimates did not expect.”

Proportional prioritization samples in proportion to each transition’s error-derived priority.
Rank-based prioritization orders transitions by priority and samples according to rank, reducing the influence of extreme errors.
Importance-sampling weights reduce the bias introduced by non-uniform sampling, so frequently selected transitions do not distort updates too severely.

Why the extra focus helps—and needs care

Imagine an agent has learned an easy route through a maze, then discovers a rare shortcut with an unexpectedly high reward. That transition produces a large TD error; prioritized replay revisits it quickly, spreading its lesson backward through earlier decisions faster than uniform replay would. This is especially valuable when rewards are sparse, as in Atari games where meaningful events are rare.

The same mechanism can amplify noise. A transition with a large error because of randomness, an outlier reward, or unstable early predictions can be replayed excessively. Priorities are therefore updated as learning proceeds, small baseline priorities ensure every experience remains eligible, and importance weighting is gradually strengthened. Prioritized replay is a key DQN refinement, used in systems such as Rainbow DQN: it makes limited interaction data more valuable, while preserving enough diversity that the agent does not learn only from its loudest mistakes.

Prioritized Experience Replay is a replay-buffer strategy that samples past transitions with probabilities weighted by their learning value, typically their temporal-difference error, rather than uniformly. Transitions whose predicted value differs most from the observed update are replayed more frequently, with importance-sampling corrections to reduce bias. It improves data efficiency and speeds value learning by focusing updates on surprising or poorly learned experiences.

Imagine studying for a test by revisiting the questions you got most wrong, rather than rereading every page equally. Prioritized Experience Replay gives an AI learner a similar habit.

As it tries actions, the AI keeps a memory of past situations and what happened next. Instead of practising those memories at random, it pays extra attention to the surprising or costly ones—moments where its expectations were badly wrong. A robot that crashes into a wall, for example, should learn more from that mistake than from another routine trip down an empty hallway. This helps it improve faster by spending more time on experiences with the most to teach.