Ranking Calibration
Suppose 70% of what you watch is comedy and 30% is documentary. A recommender that hands you ten comedies has arguably maximised relevance — every item is something you like — and has still got it wrong. Ranking calibration is the requirement that a list's composition reflect the composition of the user's interests, not only its strongest mode.
Two meanings, both in use
- Calibrated recommendation — the genre or topic mix of the list matches the user's historical mix, measured as a divergence between the two distributions.
- Score calibration — a predicted click probability of 0.2 really means one click in five, which is what makes scores from different models comparable and lets expected value be computed in auctions and multi-objective ranking.
How it is achieved
Both forms are applied after the ranking model rather than inside it. Score calibration fits isotonic regression or Platt scaling on held-out data, undoing the distortion that negative sampling introduces into predicted probabilities. List calibration is a re-ranking pass: greedily select items to maximise relevance minus a penalty on divergence from the target mix — the same machinery as diversification, with a different objective.
Why it matters
Uncalibrated lists amplify whatever the user does most, a direct route to the filter bubble: minority interests vanish from a feed even though the model scored them well. And because ranking models train on sampled negatives and heavy class imbalance, their raw outputs are not probabilities — treating them as such quietly corrupts every decision made by combining scores.
Ranking calibration covers two related corrections: making the composition of a recommended list match the user's actual interest mix, and making predicted scores behave as true probabilities. Both are applied after the ranking model — by re-ranking against a target distribution, or by isotonic regression on held-out data. Without it, lists collapse onto a user's dominant interest and scores cannot be compared or combined.
Think of someone packing your lunch. You eat sandwiches most days and fruit now and then. If they notice you like sandwiches best and pack five sandwiches, they have technically given you only things you like — and still got the balance wrong.
Ranking calibration is about keeping that balance right. If a third of what you watch is sport, roughly a third of your suggestions should be sport, instead of the software doubling down on your single biggest habit. It is how a feed keeps showing you your smaller interests rather than quietly dropping them, and it is what makes recommendations feel like you instead of a caricature of you.