Diversity Metric
Two recommendation lists can score identically on precision and feel completely different to the person reading them — one spans five genres, the other is five variations of a single film. A diversity metric is how that difference gets measured instead of guessed at.
What it computes
It is a list-level quantity, not a per-item one. The usual form is intra-list diversity: the average pairwise distance between the items in one recommended list, where distance comes from item embeddings, genre vectors or taxonomy overlap. Averaged over users, it yields one number per model. Companion measures look across users rather than inside a list — catalogue coverage, the share of items that ever get recommended to anyone, and Gini or entropy over how often each item is recommended.
Why it matters
Accuracy metrics such as NDCG and Recall@k are scored against logged relevance, and logged relevance is itself skewed towards whatever the previous policy chose to show. A model can improve on those metrics while quietly collapsing onto the head of the catalogue, and nothing in the accuracy numbers reveals it:
- It makes the relevance-versus-variety trade-off reportable, so a re-ranking parameter can be tuned against a number rather than a hunch.
- It exposes popularity collapse early — coverage falls before engagement does.
- It is cheap and entirely offline: no experiment and no traffic, just the lists the model already produces for a held-out set of users.
- It is never an objective on its own. A list of random items maximises diversity, which is why it is always read alongside an accuracy metric and a beyond-accuracy measure like novelty.
A diversity metric quantifies how varied a recommendation list is rather than how accurate it is. The common form, intra-list diversity, averages pairwise distance between recommended items using embeddings or attributes; catalogue coverage measures the same concern across users. It makes the relevance-versus-variety trade-off measurable offline and exposes a model collapsing onto popular items while accuracy still improves. It is read alongside accuracy, never alone — a random list maximises it.
Imagine two playlists that both suit your taste. One moves between folk, jazz and electronica. The other is ten songs by one band. A diversity metric is simply a way of putting a number on that difference.
It looks at a list the software produced and measures how unlike each other the items in it are — and sometimes how much of the whole catalogue ever gets shown to anybody at all. Why bother? Because a system can look excellent on paper while quietly serving everyone the same few popular things. Measuring variety is how that gets caught. On its own it proves nothing: a list of random junk would score beautifully, so it is always read next to how well the suggestions actually land.