Exploration-Exploitation (RecSys)
Every recommendation slot is a small bet. Showing the item your model already ranks highest collects a reliable reward; showing one it is unsure about collects information instead. Exploration-exploitation is the name for that trade, and in recommendation it is forced rather than optional.
Why the choice is forced
A recommender only observes feedback on what it chose to show. An item never surfaced earns no clicks, so a low estimate of it is never corrected, and the logs simply confirm the policy that produced them. Pure exploitation freezes the catalogue around whatever looked good early, leaving new items and new users permanently unmeasured. Pure exploration burns impressions on items already known to be poor.
How systems strike the balance
The practical recipes come from the bandit setting, applied one slot at a time:
- ε-greedy reserves a small share of impressions for under-served items — cheapest to build and easiest to audit.
- Upper confidence bound adds an uncertainty bonus to each estimate, so an unproven item gets a boost that shrinks as evidence arrives.
- Thompson sampling draws from the posterior over each item's reward and plays the winner, spreading exploration in proportion to genuine doubt.
Why it matters
Exploration is the only mechanism that stops a ranking policy becoming a self-fulfilling record of its own past. It is also what makes later measurement possible: logged impressions with known selection probabilities are exactly what inverse propensity scoring needs to estimate how a different model would have performed. The cost is immediate — a deliberately sub-optimal slot today, in exchange for a catalogue the system can still learn about tomorrow.
Exploration-exploitation is the trade-off between serving the item a model already scores highest and serving one whose value is uncertain in order to learn about it. A recommender observes feedback only on what it displays, so pure exploitation locks the catalogue into its own past and starves new items and new users of evidence. Bandit policies such as ε-greedy, upper confidence bound and Thompson sampling allocate that balance slot by slot.
Think of a favourite restaurant. You could order the dish you know you love every single time, or occasionally try something new. Order the usual forever and you will never discover a better dish. Gamble every visit and most evenings will disappoint.
Apps that choose what to show you face the same decision with every slot on the screen. Playing your safe favourite is exploiting what the system already knows about you. Slipping in something unproven is exploring. Exploring costs a little, because you might not like it. But it is the only way the app ever learns about the songs, films and products it has never shown anyone — including the ones you would have loved.