Contextual Bandit (RecSys)

A plain bandit learns which single arm pays best. A recommender cannot use that answer, because the best item depends entirely on who is asking. A contextual bandit is the version that gets to see who is asking before it chooses.

The loop, and what recommendation adds to it
On each request the policy receives a context vector — the user's embedding, their recent interactions, the time, the device — picks an item from the candidate set, and observes a reward for that item only. Everything it did not show stays unobserved. LinUCB keeps a linear reward model with a confidence bonus; Thompson sampling draws from a posterior over the reward parameters and acts greedily on the draw. What the recommendation setting adds is scale and churn: the arms are catalogue items, millions of them, arriving and dying continuously, so a per-arm model gives way to one shared model over item features.

Where it earns its keep

A new item has no logged reward at all. The confidence bonus or posterior variance gives it exposure for a stated reason, rather than through a hand-tuned rate of random injection.
Non-stationary catalogues — a news feed where yesterday's winner is today's stale story — need a policy that keeps re-measuring rather than one retrained nightly.
It records the probability with which it chose each item, which is exactly the propensity that off-policy evaluation needs later.

Without it
A purely greedy ranker only ever learns about what it already shows. Its own logs then confirm its own choices, popularity bias hardens, and the part of the catalogue it never surfaces becomes permanently unmeasurable.

A contextual bandit chooses an item per request after observing a context — user features, recent behaviour, time, device — and learns from the reward of that single choice, everything unshown staying unobserved. LinUCB and Thompson sampling are the standard policies. It gives a recommender a principled way to spend exposure on items it is uncertain about, which breaks the feedback loop that otherwise entrenches whatever the model already favours.

Picture a waiter with a new menu. He can keep bringing you the dish you ordered last time and be safely right. Or he can occasionally suggest something he is unsure about, and actually find out whether you like it.

A contextual bandit is software making that trade on purpose. The context is simply what it knows at the moment of asking: the time, the device, what you watched yesterday. It picks something, sees whether you engage, and updates its guess. The deliberate uncertainty is the clever part. A system that only ever shows its safest option never learns anything about the rest of what it has, and quietly shrinks to the same few hits for everyone.