Reinforcement Learning for Recommendation
A ranker trained to predict the next click optimises one moment. A system trained with reinforcement learning optimises the session — and the weeks after it. The question shifts from "which item gets clicked now" to "which item leads somewhere worth going".
The sequential framing
Recommendation becomes a decision process: the state is what is known about the user so far, the action is the item or slate shown, and the reward is the engagement that follows. A policy maximises discounted cumulative reward instead of immediate accuracy, so an item with modest click probability is still the right action if it opens a region of the catalogue the user keeps returning to. A contextual bandit is the one-step corner of this setting, where the action is assumed not to change the next state.
What makes it hard here
- The action space is the catalogue, and slates make it combinatorial. SlateQ decomposes a slate's value into per-item terms to stay tractable.
- Training is off-policy: the logs come from an earlier policy, so importance weights — as in the top-K off-policy correction applied to REINFORCE at YouTube — carry the fix, and its variance.
- Reward is delayed. Watch time, return visits and retention arrive long after the action that earned them.
Why it matters
Greedy next-click ranking manufactures the failure modes the field complains about: narrowing feeds, entrenched popularity, no incentive to learn about an item never shown. A long-horizon objective makes exploration and the user's changing state part of what is being optimised rather than a correction bolted on afterwards.
Reinforcement learning for recommendation frames the problem as a sequential decision process: the state is the user's history, the action is the item or slate shown, the reward is the engagement that follows. A policy is trained to maximise long-horizon return rather than immediate click probability, learned off-policy from logged interactions with importance corrections. It matters because greedy next-click ranking ignores how today's list changes tomorrow's user, leaving session value, retention and exploration outside the objective.
Imagine a shop assistant who can either sell you the one thing you were obviously going to buy today, or spend a minute showing you something that turns you into a regular. Taking the easy sale every single time is the cheaper strategy and the worse one.
Reinforcement learning is how recommendation software learns to think past the next click. Instead of asking only what you will tap right now, it tries to pick things that make the whole visit, and the next one, go well. That means occasionally showing you something unproven, because finding out what you actually like is worth something too.