Inference Latency Budget
A recommendation that arrives after the page has finished rendering is worth nothing. Long before anyone argues about model architecture, serving fixes a ceiling — a few tens of milliseconds — and every stage of the pipeline has to fit inside it.
A budget is divided, not simply spent
The latency budget is the end-to-end allowance for producing one list, split across the whole path: feature lookups from an online store, candidate generation over the catalogue, the ranking pass, re-ranking and business rules, plus network time. A 50 ms budget might leave 10 ms for feature fetches, 10 for retrieval and 20 for scoring a few hundred candidates. It is enforced at the 99th percentile, not the average — the tail is what users feel and what trips timeouts.
What the budget decides for you
Most architectural choices in a production recommender are this one constraint in disguise.
- The retrieve-then-rank split exists because an expensive model cannot see the whole catalogue in time.
- Two-tower models win retrieval because item vectors are precomputed and searched with FAISS, leaving only the user tower to run per request.
- Cross features and deep interaction models live in ranking, where the candidate set is already small.
When it is ignored
A model that overruns its slice does not simply get slower: the serving layer drops it and falls back to a cached or popularity-based list. Offline metrics then look excellent while the experiment shows nothing, because a share of requests never reached the model at all.
An inference latency budget is the end-to-end time a recommender is allowed to spend producing a list, divided across feature lookup, candidate generation, ranking and re-ranking, and enforced at tail percentiles rather than on average. It is the constraint that forces the retrieve-then-rank architecture and precomputed item embeddings. Overrun it and the serving layer falls back to a cached or popular list, so model gains never reach the user.
Think of a kitchen that has promised your meal in ten minutes. Everything on the menu has to be cookable in that time, which quietly rules some dishes out no matter how good they taste.
A recommender works the same way. From the moment you open an app, it has a few hundredths of a second to decide what to show you, and that allowance is its latency budget. The time gets shared out: looking up who you are, shortlisting a few hundred candidates from millions, then putting those in order. If a cleverer model cannot finish inside its share, the system gives up on it and shows you something generic instead.