Inverse Propensity Scoring (IPS)
Your logs record what happened while the old ranker was in charge. What you want to know is what would happen with a new one. The data is not neutral evidence — it is a record of one policy's choices — and inverse propensity scoring is the correction that makes it usable anyway.
Weight each event by how unlikely it was
Every logged impression had some probability of being shown, its propensity. Inverse propensity scoring estimates a new policy's value by averaging the rewards of logged events, each divided by that probability: an item the old system showed 90% of the time counts for little, while one it showed 1% of the time counts a hundred times as much. The reweighted sample behaves like one drawn without the old policy's preferences, which is why the estimate is unbiased — provided every item had a non-zero chance of being shown and the propensities were actually logged.
The variance problem
Dividing by a small number is dangerous. A single click on a rarely shown item can dominate the whole estimate, so raw IPS is unbiased but wild. Practice clips the weights at a ceiling, trading a little bias for stability, or moves to doubly robust estimation, which subtracts a model's prediction and reweights only the residual.
Not only for evaluation
The same weights appear in training: unbiased learning to rank divides each example by its estimated examination propensity so the model stops learning that the top of the list is intrinsically good. Without this correction, both offline evaluation and training reward whatever the logging policy already preferred.
Inverse propensity scoring estimates how a new policy would have performed using logs a different policy generated, by weighting each logged event by the inverse of its propensity — the probability the logging policy had of showing it. The reweighting cancels the old policy's selection bias and makes the estimate unbiased. Rare events carry huge weights, so the estimator has high variance and is normally clipped or replaced by a doubly robust version.
Suppose you want to know whether a different shop assistant would have sold you more. All you have is a record of what the old assistant chose to show you — and he almost always reached for the same few products.
Inverse propensity scoring is the arithmetic that fixes this. Each thing you were shown is counted in proportion to how rare it was to be shown it: the usual suggestions count for little, the unusual ones count for a lot. That rebalancing lets a company judge a new recommendation system from old records, without having to try it out on real customers. Its weakness is that rare cases carry so much weight that one of them can distort the answer.