Off-Policy Evaluation
Imagine having logs from yesterday’s decisions—what actions were taken, what happened, and what reward followed—but being unable to run new experiments. Off-policy evaluation asks a crucial question: how well would a different decision policy have performed using only those old records?
What is being estimated
The recorded data was generated by a behaviour policy: the policy that actually selected actions. The policy being judged is the target policy. OPE estimates the target policy’s expected long-term return without deploying it in the environment. This differs from ordinary policy evaluation, where the policy being evaluated is also the one that generated the experience.
How it works—and its hard limit
A basic method, importance sampling, reweights each logged outcome by how much more or less likely the target policy was to choose the recorded actions. More advanced estimators such as doubly robust evaluation combine those weights with a learned value model, reducing variance when either component is reliable. OPE requires coverage: actions the target policy relies on must appear in the logged data with nonzero probability. Without coverage, the data contains no evidence about those choices, and no estimator can reliably fill in the gap.
Why it matters in reinforcement learning
Live trial-and-error can be expensive, slow, or unsafe. Before deploying a controller trained in simulation, for example, OPE can test whether its choices appear promising in real logged trajectories. It is also valuable when selecting among policies trained with DQN or PPO: a policy that looks strong on training returns can fail when dynamics shift slightly. Poorly configured OPE can be dangerously overconfident: importance weights can explode when the target policy differs greatly from the behaviour policy, while a biased value model can systematically favor an unsafe shortcut. Sound OPE therefore provides a bridge between logged experience and cautious policy improvement.
Off-policy evaluation estimates the expected return of a target policy using data collected by a different behavior policy, without deploying the target policy in the environment. Methods such as importance sampling correct for the mismatch in action choices. It matters because it enables safer, cheaper policy comparison and selection from logged experience, while controlling the bias and variance introduced by distribution mismatch.
Imagine judging a new restaurant menu using only receipts and customer reactions from the old menu. You cannot serve the new dishes yet, but you still want a sensible estimate of whether people would like them.
Off-policy evaluation does this for a decision-making AI. It estimates how well a new strategy would perform by studying data collected while a different strategy was in charge. For example, a recommendation system might use past clicks to estimate whether a new set of recommendations would help, before showing it to real users. This matters because testing every new idea live can be expensive, risky, or unfair to people.