Bandit-Based Recommendation
Every slot you fill is two things at once: a chance to please someone now, and a chance to learn something you do not yet know. You cannot have both from the same impression, and that tension is what bandit-based recommendation is built to manage.
The framing
Each candidate item is an arm. Showing it is pulling that arm, and the reward is whatever the user does next — a click, a play, a purchase. The crucial property is that feedback is bandit feedback: you learn the outcome of the item you showed and nothing at all about the items you did not. A supervised model trained on those logs therefore learns only about the slice its own past decisions chose to reveal.
How a policy decides
The useful algorithms act on uncertainty rather than on the point estimate alone:
- Upper-confidence-bound methods add a bonus that grows with how little an item has been shown, so untried items get a turn on their own merits.
- Thompson sampling draws one value from each item's posterior and shows the winner, which spends exploration in proportion to genuine doubt.
- Contextual variants condition all of this on user and item features, so the policy generalises across users instead of learning each item from scratch.
Where it sits and why it matters
The bandit usually governs the final step — which of the top-scored candidates actually occupies the slot — and it logs the probability with which each choice was made, which is what later makes off-policy evaluation possible. Its real contribution is breaking the feedback loop: a purely greedy ranker never shows a new item, never collects evidence for it, and so confirms its own prior forever. A bandit pays a small, budgeted cost in short-term click-through and gets item cold start resolved in hours.
Bandit-based recommendation treats each impression as choosing one arm from a set and observing only that choice's reward, so the policy must balance showing what currently scores best against gathering evidence on what it has barely shown. Upper-confidence-bound and Thompson-sampling policies, usually conditioned on context, do this explicitly. It matters because a purely greedy ranker trains on its own logs and never surfaces new items, leaving cold start and popularity bias to reinforce themselves.
Imagine a café with twenty cakes in the window but only one slot on the counter. You could keep putting out the cake that sold well last week and be fairly safe. Or you could occasionally put out one nobody has tried, and find out whether it is a hit. Play it safe forever and the new cakes never get a chance.
Bandit-based recommendation is software making that choice deliberately. Each thing it could show you is an option whose appeal it only partly knows. Most of the time it shows what looks best, and now and then it tries something uncertain, on purpose, to learn. That small sacrifice is how anything new ever gets discovered.