Embedding Index

A two-tower model hands you a user vector at request time and a vector for every item in the catalogue. The obvious next move — compare the user against all of them — is the one you cannot afford. An embedding index is the structure that answers which items are nearest a vector without touching most of them.

What it is
Item vectors are computed offline by the item tower and loaded into an approximate nearest neighbour structure: an inverted file of centroids that probes only the closest clusters, a navigable small-world graph such as HNSW that walks greedily towards the query, or product quantisation that compresses vectors so more fit in memory. FAISS, ScaNN and hosted vector databases package these. In practice the index is a build artefact — rebuilt or incrementally updated on a schedule — and the user tower is the only part that runs per request.

Why it matters
The index is what makes the retrieval stage of a multi-stage recommender possible at all: a few milliseconds from one vector to a few hundred candidates out of millions.

Its recall is a tunable approximation. Probe fewer clusters and candidates arrive faster and worse, and the ranker can never recover an item retrieval missed.
It must be rebuilt whenever the item tower is retrained, because vectors from two model versions are not comparable; a stale index degrades relevance silently.
New items join by encoding and insertion rather than retraining, which is why embedding retrieval absorbs item cold start better than id-only models.

An embedding index is the approximate-nearest-neighbour structure holding every item's vector so a user vector can be matched against a huge catalogue in milliseconds. Built offline from the item tower using methods such as inverted-file clustering, HNSW graphs or product quantisation in libraries like FAISS, it is what makes the retrieval stage practical. Its recall caps the whole pipeline — items it misses never reach the ranker — and it must be rebuilt whenever the model changes.

Picture a library with ten million books and no catalogue. Finding the three closest to your taste would mean reading the lot. An embedding index is the catalogue — prepared in advance so the search can jump to the right shelf instead of checking every book.

Recommendation systems describe each item with a short list of numbers, then store all those descriptions in this index. When you show up, your own list of numbers is matched against it and a few hundred likely candidates come back in a fraction of a second. Because it takes a shortcut, it can occasionally overlook something you would have loved — and once it does, nothing later in the process can put that item back.