Off-Policy Learning

Imagine learning how to drive from a mixture of your own trips, a cautious instructor’s recordings, and old driving logs. Off-policy learning lets a reinforcement-learning agent learn the best decision rule from data generated by a different decision rule. That separation is powerful because collecting experience is expensive, risky, and necessarily exploratory.

Two policies, two jobs
The behaviour policy chooses actions and produces the data: it might take random actions, follow a safety-focused controller, or come from a saved dataset. The target policy is the policy the algorithm wants to evaluate or improve. In off-policy learning, these need not match. For example, Q-learning learns action values using the update target r + γ maxₐ′ Q(s′, a′): it asks what the best next action would be, even when the recorded next action was exploratory.

Why the separation matters
It enables several practical capabilities:

• Reuse: replay past transitions many times from a replay buffer, rather than discarding them after one update.
• Safe data collection: learn a stronger policy from demonstrations or a conservative data-gathering policy without deploying the still-untrained agent.
• Exploration without commitment: collect diverse trials while learning a greedier policy that exploits what those trials reveal.

The catch: correcting a data mismatch
Data from the behaviour policy can misrepresent what the target policy would encounter. Methods such as importance sampling reweight samples by how differently the two policies would choose an action; value-based methods like DQN instead use bootstrapped value targets and replay buffers. This flexibility also creates instability: the combination of function approximation, bootstrapping, and off-policy data can cause value estimates to drift or diverge. DQN’s target network and replay buffer were designed to make that combination more stable. In a Gymnasium task, this is why DQN can learn from a large, mixed archive of transitions, whereas on-policy methods such as PPO mainly require freshly collected data from their current policy.