Diversification (Ranking)
A ranker that scores every candidate on relevance alone will happily hand you ten slots filled with the same thing — ten episodes of one series, ten lenses for one camera body. Each is individually the best prediction, and together they make a worse list. Diversification is the step that trades a little predicted relevance for variety across the list.
How it is done
It runs as a re-ranking pass over candidates that have already been scored, so the objective shifts from scoring each item to scoring the set. The standard device is greedy marginal-gain selection: take the top item, then repeatedly add the candidate that maximises a blend of its own score and its dissimilarity to what is already chosen. Maximal Marginal Relevance is the canonical form, with one knob trading relevance against redundancy; determinantal point processes express the same idea as a volume in embedding space. Similarity comes from the item embeddings or attributes — genre, brand, seller — the ranker already holds.
Why it matters
Pointwise and pairwise ranking losses are blind to redundancy, because they never see two candidates as a list. Diversification is where set-level quality is added back:
- It hedges a wrong relevance estimate — ten guesses on one theme fail together, a varied slate does not.
- It supplies the exploration the next training round learns from, limiting the popularity bias and narrowing feedback loops that relevance-only ranking compounds.
- Its cost is bounded: it touches only the few hundred candidates that survive retrieval, so it fits the serving latency budget.
- Without it, offline accuracy looks healthy while the list reads as repetitive and engagement drifts down.
Diversification is a re-ranking step that trades a small amount of predicted relevance for variety across a recommendation list, so the final slate is not ten near-duplicates. It scores the set rather than each item, usually by greedy maximal marginal relevance over the candidates that survive retrieval. Ranking losses are blind to redundancy, so without it lists collapse onto one theme, exploration stops and popularity bias compounds.
Think of a shop assistant who has worked out that you like running shoes, and then shows you eleven pairs of almost exactly the same shoe. Each one is a fair guess. As a shelf, it is useless.
Diversification is the step that stops that happening. Once the software has ranked everything it could put in front of you, it deliberately swaps some of the near-identical picks for choices that still suit you but differ from each other — another brand, another style, something next door to your taste. It gives up a sliver of certainty so that if the guess is wrong, the whole list does not fail at once.