Selection Bias (RecSys)
A recommender's training data is not a sample of what users think about items. It is a record of what a previous system decided to show, filtered by what users then chose to react to. Selection bias is the gap between those two things.
Where it comes from
Two mechanisms compound. The logging policy decides which user-item pairs ever get an impression, so an item never shown generates no data and can never be learned about. Then the user decides which impressions to act on, and that choice correlates with preference — people rate films they chose to watch, which is why the observed rating distribution on MovieLens skews high. The result is that missingness is not at random: whether a cell of the user-item matrix is observed depends on its value.
What it breaks
- Evaluation. Averaging a metric over logged interactions estimates performance on the old policy's choices, not on the catalogue, which is why offline gains fail to reproduce online.
- Training. A model fitted on observed cells alone learns the logging policy's preferences and reinforces them — the mechanism behind popularity bias and feedback loops.
What to do about it
The standard correction treats being shown an item as a treatment and reweights by the inverse of its probability — inverse propensity scoring, with doubly robust estimators to tame the variance when propensities are small. The gold standard is a small randomised-exposure slice of traffic, uniformly sampled, which yields an unbiased test set and the propensity estimates everything else depends on.
Selection bias is the fact that logged interaction data covers only user-item pairs a previous policy chose to show and the user chose to act on, so the user-item matrix is missing not at random — whether a cell is observed depends on its value. It biases both training and offline evaluation toward the logging policy, driving popularity feedback loops. Corrections use inverse propensity scoring, doubly robust estimators, or a randomised-exposure slice of traffic.
A restaurant's reviews only tell you about the dishes people ordered. If one dish is hidden at the bottom of the menu, it collects no reviews — not because it is bad, but because almost nobody saw it.
Selection bias is that same problem inside a recommender. It only ever learns from things it already chose to put in front of you, and from the handful of those you bothered to click. So items it never displayed stay unknown indefinitely, popular things look even better than they are, and the system can keep confirming its own habits while believing it has evidence.