Sampled Metrics

Scoring a ranking metric honestly means ranking the held-out item against the whole catalogue. Doing that for every test user is expensive, so a shortcut became standard: rank it against a handful of random items instead. The shortcut has a well-documented cost.

The protocol
For each test interaction, take the one held-out item plus typically 99 or 100 items the user has not interacted with, score that small candidate set, and compute Hit Rate, NDCG or MRR on it. These are sampled metrics. Full evaluation instead scores every unseen item — hundreds of thousands of them — for every user.

Why the two disagree

Sampled metrics are not consistent estimators of their full-catalogue counterparts. The mapping from true rank to expected sampled rank is non-linear, so the ordering of two models can invert.
The distortion is largest at small sample sizes and for exactly the top-heavy metrics people report, which is where published rankings of sequential models have flipped under re-evaluation.
Negative sampling is almost always uniform, which over-represents the long tail and makes a popularity baseline look weaker than it is.

What to do with them
Treat a sampled number as comparable only with another sampled number produced by an identical sampling scheme, and never across papers. Where the catalogue allows it, evaluate fully — scoring a million items per user is a matrix multiply, not a research problem — or use corrected estimators that de-bias the sampled rank before averaging.

Sampled metrics compute ranking measures such as Hit Rate or NDCG against the held-out item plus a small set of sampled negatives — typically a hundred candidates — instead of the full catalogue. The shortcut cuts evaluation cost, but it is not a consistent estimator of the full-catalogue metric: it compresses rank differences non-linearly and can reverse the ordering of two models. Numbers are comparable only under an identical sampling scheme.

To test whether a recommender can find the one thing you wanted, you could make it search the entire catalogue — millions of items. That is slow, so researchers often make it choose between the right answer and ninety-nine random wrong ones instead.

That is a sampled metric, and it flatters everybody. Picking the right film out of a hundred is far easier than picking it out of a million, and the easy test does not put systems in the same order as the hard one. So a score measured this way can only be compared with another score measured in exactly the same way.