Mean Reciprocal Rank (MRR)
Some recommendation problems have exactly one right answer — the next track the listener actually played, the product they went on to buy. For those, the only question worth asking is how far down the list it sat. Mean reciprocal rank turns that distance into a score.
The calculation
For each user or session, find the rank of the first relevant item and take its reciprocal: rank 1 scores 1, rank 2 scores 0.5, rank 4 scores 0.25. Average across all cases and you have MRR; a session whose target never appears in the truncated list contributes zero. The decay is deliberately harsh — lifting a hit from position four to position two is worth more than lifting it from twenty to ten, which matches the way attention falls away down a page.
When to use it, and when not
- It fits single-target tasks. Sequential models such as GRU4Rec and SASRec are reported with MRR@10 and Hit Rate@10 side by side, because next-item prediction has one held-out answer per session.
- It ignores everything after the first hit, so for a user with ten relevant items it discards nine of them — MAP or NDCG is the better choice there.
- Graded relevance is outside its vocabulary: every relevant item counts the same.
A caveat worth carrying
MRR is usually computed against sampled negatives — the true target plus ninety-nine random items — because scoring the full catalogue for every test case is expensive. Those numbers run far higher than full-catalogue values and do not reliably preserve the ranking between models, so an MRR is only comparable with another computed the same way.
MRR is the average of the reciprocal of the rank at which the first relevant item appears, so a hit at position one scores 1, a hit at position four scores 0.25, and a miss scores zero. It suits single-target tasks such as next-item prediction, where only the first correct hit matters. It ignores any further relevant items, which makes MAP or NDCG the better measure when a user has several valid targets.
Suppose you ask a friend to name the film you are in the mood for. If they get it first guess, that is impressive. Third guess is fine. Fifteenth guess is not much help at all.
Mean reciprocal rank is a score for exactly that behaviour. It looks only at where the right answer first turned up in the list, gives full marks for the top spot and rapidly less the further down it sits, then averages over many attempts. It is the natural score when there is one correct answer — the song you actually played next, say — rather than a whole set of things you might enjoy.