Online Recommendation
The moment a user taps an item, something about them has changed — and a list computed last night cannot know it. Online recommendation is the alternative: candidates are retrieved and scored inside the request itself, so what gets shown reflects the session as it actually is.
What happens in those milliseconds
The request arrives, features are fetched from a low-latency store, a retrieval step pulls a few hundred candidates — commonly a two-tower model's user embedding queried against an index like FAISS — a ranker scores them, and a re-ranking pass trims the slate. The whole path has a budget, often under 100 milliseconds, and it shares that budget with everything else the page needs.
What it makes possible
- Session signal — the last three items viewed sit in the feature vector, which is the only way sequential models like GRU4Rec or SASRec earn their keep.
- Real-time context — device, time of day, what is currently in stock or already in the cart.
- New users — somebody who signed up a minute ago still gets scored, where a batch job has no row for them.
The cost side
Every request runs model inference, so capacity scales with traffic rather than with the size of the user base, and the latency budget caps the model you can afford. That is exactly why production systems push the heavy work into retrieval indexes and precomputed embeddings and leave only the ranking pass truly online.
Online recommendation retrieves and scores candidate items during the user's request rather than in a precomputed batch, so the result reflects the current session, live context and freshly created users. It is what makes sequential and context-aware models usable, and it is bounded by a latency budget — typically tens of milliseconds — which limits model size and forces the heavy work into retrieval indexes and cached features.
Picture a waiter who watches what you have already eaten and adjusts the next suggestion on the spot, instead of handing you a menu printed last week. Online recommendation is that. The app works out what to show you at the moment you ask, taking in what you clicked a minute ago. It is why a feed can shift during a single sitting, and why someone who joined today still gets sensible suggestions. The catch is time. All of that thinking has to finish before the page appears, so it has to be quick — and the computer has to do it again for every single visitor.