Negative Sampling for Evaluation
Ranking one held-out item against an entire catalogue, for every user in the test set, is expensive. The shortcut is to rank it against a handful of items the user never touched — classically 99 sampled negatives plus the one true positive. Negative sampling for evaluation is that shortcut, and it is the most criticised habit in recommender benchmarking.
How it is done
For each held-out interaction, draw N items the user has not interacted with, score the resulting 100-item set, and compute Hit Rate@10 or NDCG@10 over it. The draw is usually uniform across the catalogue, sometimes popularity-weighted. The MovieLens protocol published alongside NCF made 99 uniform negatives the de facto standard, which is why so many reported numbers are comparable only to each other.
Why it distorts
Uniform draws from a long-tail catalogue mostly return obscure items, so the task is far easier than the real one and absolute scores inflate. The damage is not uniform across models: sampling compresses the very differences a top-k metric exists to expose, and it can reorder which model looks best — so a conclusion drawn on sampled metrics need not survive on the full catalogue. Krichene and Rendle’s analysis of sampled metrics is the standard reference.
What to do instead
Evaluate against the full catalogue when it fits, which for most public datasets it does — scoring is one matrix product. If sampling is unavoidable, state the sampler and the count, and read the result as a relative signal inside one protocol, never as a portable figure.
Negative sampling for evaluation scores a held-out interaction against a small set of items the user never engaged with — commonly 99 sampled negatives — instead of the full catalogue, which makes top-k metrics cheap to compute. It also makes them unreliable: uniform draws from a long-tail catalogue inflate the scores and can change which model ranks best, so sampled results compare only within an identical protocol.
Imagine testing whether a friend really knows your taste in films. You could show them your favourite alongside every film ever made and ask them to pick it out — or alongside ninety-nine films pulled at random. The second test is much quicker, and much easier to pass, because most random films are obviously wrong.
Negative sampling for evaluation is that shortcut, used to grade recommendation software. It saves enormous amounts of computing time, but it flatters the system being tested and can even make a weaker method look like the winner. Results from it only mean something next to other results measured in the exact same way.