Counterfactual Evaluation
Your logs record what happened while the current recommender was in charge. The question you actually want answered is what would have happened under a different one — and that experiment never ran. Counterfactual evaluation is the machinery for answering it anyway.
The missing-data problem at the heart of it
Logged feedback is not a random sample of the catalogue. A user engaged, or declined to engage, with the items the logging policy chose to show, and nothing was recorded about everything else. Scoring a new policy against those logs naively rewards it for agreeing with the old one. Counterfactual evaluation treats the log as a biased sample instead, dividing each logged outcome by the probability the logging policy had of showing that item — its propensity — so frequently-shown items stop dominating the estimate. That is inverse propensity scoring; doubly robust estimators add a reward model to cut its variance.
What it buys and what it demands
- It replaces weeks of live traffic with an offline calculation over logs you already hold, and exposes no user to a policy that turns out to be bad.
- It needs the propensities, so the logging policy must have been stochastic and must have recorded its own action probabilities. A deterministic ranker leaves logs that cannot support this at all.
- Estimates degrade as the new policy strays from the old one: rare events pick up enormous weights and variance explodes. Clipping those weights trades a little bias for a usable number.
Why it matters here
This is the honest answer to the gap between offline metrics and online results. Skip it and the only way to learn whether a policy works is to ship it to real users and watch.
Counterfactual evaluation estimates how a new recommendation policy would have performed using logs produced by a different policy, correcting for the fact that users only ever saw what the old system chose to show. It makes offline comparison of candidate rankers meaningful rather than self-confirming, so a weak model can be rejected before an A/B test spends traffic on it. Without the correction, logged clicks simply reward whatever the logging policy already preferred.
Imagine a shop that only ever put six products in the window, and now wants to know whether a different six would have sold better. It cannot rewind the week — it only has sales figures for the choices it actually made. Counterfactual evaluation is the set of techniques that answer that question anyway.
It takes the records of what people clicked or bought, works out how likely the old system was to show each thing, and reweights the history so a new system can be scored fairly on it. Without that step, past records simply flatter whatever was already being shown.