Interleaving
Two rankers differ by a little, and an A/B test says it will take three weeks and a million users to tell whether the difference is real. Interleaving asks a sharper question: shown both lists at once, which one does this single user pick from?
One list, two authors
Instead of giving ranker A to one group of users and ranker B to another, interleaving merges their outputs into a single list and records which ranker contributed each clicked item. Team-draft interleaving alternates picks the way two captains choose a team, with a coin flip deciding who picks first; probabilistic variants sample from each ranker's own distribution so credit can be attributed without distorting either list. The winner is whichever ranker's items get clicked more.
Why it is so much more sensitive
Every user becomes their own control, which removes the variance that dominates an A/B test — different people click at wildly different rates, and that noise is exactly what interleaving cancels. Comparisons at web-search scale report the same verdict as a split test from one to two orders of magnitude less traffic, turning a three-week experiment into a one-day one.
What it cannot tell you
It measures a preference between two rankings, not a business outcome: it will not give you revenue, session length, or the effect of showing more recommendations. So it is used as a fast screen — cheap enough to run on every candidate ranker — with a conventional A/B test reserved for the few that survive it.
Interleaving compares two rankers by merging their outputs into one list shown to a single user and attributing each click back to the ranker that contributed the item. Because every user acts as their own control, it detects small ranking differences on a fraction of the traffic an A/B test needs. It measures ranking preference only — not revenue or session length — so it serves as a fast screen ahead of a conventional experiment.
Imagine two friends who each recommend films. Instead of following one friend's list this month and the other's next month, you shuffle their suggestions into a single list and simply notice whose picks you actually watch.
That is interleaving. When a company wants to know which of two recommendation systems is better, it blends both sets of results into one list and keeps track of which system suggested the things people clicked. Because the same person sees both, the comparison is much quicker and cleaner than splitting people into separate groups — days rather than weeks. It only answers which list people prefer, not whether the business did better.