Contextual Bandit

A contextual bandit is a decision problem where an agent sees information about the current situation, chooses one option, and receives a reward for that choice. The key challenge is learning which option works best for this particular situation, while still trying unfamiliar options enough to discover better ones.

How the decision loop works

At each round, the agent receives a context x: features describing the current opportunity. It then selects an action (also called an arm), observes only the reward from that chosen action, and moves on to a new context. It never sees the rewards that the actions it did not choose would have produced. For example, a news system sees a reader’s device, time, and recent interests, chooses one headline to display, and observes whether the reader clicks it.

Context: reader features, patient measurements, or a game’s current conditions.
Action: a headline, treatment, advertisement, or configuration choice.
Reward: a click, recovery signal, sale, score, or another immediate outcome.

Exploration with a purpose

A contextual bandit must balance exploitation—choosing the action its current model predicts will pay best—with exploration—trying actions whose value is uncertain. LinUCB does this by favoring actions with both high predicted reward and high uncertainty. Contextual Thompson sampling instead samples plausible reward models, naturally producing exploratory choices. Without exploration, a system can lock onto an early favorite and never collect evidence that another choice is better for a different type of context.

What makes it different from full reinforcement learning

In a contextual bandit, today’s action does not alter tomorrow’s context in a way the agent needs to plan around. There is no long chain of consequences: choose, receive reward, reset. Full reinforcement learning handles settings such as robot control, where an action changes the next state and a locally attractive move can cause trouble later. Contextual bandits are therefore useful when feedback is immediate, but personalization still matters—such as selecting a recommendation or allocating a promotional offer—while avoiding the unnecessary complexity of long-horizon planning.

A contextual bandit is a sequential decision problem in which an agent observes a context, selects one action, and receives only that action’s reward before the next independent decision. It must learn which actions perform best for different contexts while balancing exploration and exploitation. Contextual bandits matter because they model personalized, one-step choices—such as selecting a recommendation or advertisement—without requiring long-term state dynamics.

Imagine a music app choosing a song for you. It does not just ask, “Which song is popular?” It also notices the situation: your past listening, the time of day, or whether you are working out. That situation is the context.

A contextual bandit is a decision-maker that learns which choice tends to work best in different situations. It tries options, sees a quick result—such as whether you listened or skipped—and gradually gets better at matching choices to circumstances. This matters because one “best” choice rarely suits everyone: a comedy recommendation may delight one person and bore another.