Approximate Nearest Neighbour Retrieval
Once users and items are encoded into the same embedding space, recommending reduces to a geometry question: which item vectors lie closest to this user's vector? The catch is that answering it exactly means touching every item in the catalogue, and the request has a few milliseconds.
How the approximation is bought
Approximate nearest neighbour retrieval builds an index over the item vectors offline, then answers a query by examining a tiny fraction of them. Two families dominate. Graph indexes such as HNSW link each vector to a few neighbours and walk greedily towards the query from an entry point. Partition-and-compress indexes such as IVF with product quantisation cluster the space, scan only the nearest few clusters, and compare against compressed codes instead of full vectors. Both expose one knob — how many neighbours to explore, how many clusters to probe — that trades recall against latency.
Where it sits
This is the retrieval stage of a two-tower recommender. The item tower runs as a batch job and its outputs are loaded into the index; at request time only the user tower runs, producing a single query vector. FAISS, ScaNN and hnswlib are the implementations you will actually meet.
Why it matters
Approximation is affordable here precisely because a ranking model re-scores the few hundred items that come back, so small ordering errors wash out. What does not wash out is recall:
- An item the index never returns can never be ranked, so retrieval recall is a hard ceiling on the whole pipeline.
- A stale index leaves new items unreachable, turning an infrastructure lag into item cold start.
- Tuning for latency alone drops tail items first, since they sit in sparser regions of the space.
Approximate nearest neighbour retrieval finds the item embeddings closest to a query embedding without scanning the catalogue, using a prebuilt index — a navigable graph such as HNSW, or clustering with product quantisation — that inspects a small fraction of the vectors and trades recall for speed. It is what makes two-tower retrieval servable at catalogue scale: whatever the index fails to return can never be ranked, so its recall caps the entire pipeline.
Imagine a library with ten million books and no catalogue. Finding the five closest to your taste by pulling every book off the shelf would take forever. A librarian who has arranged the shelves sensibly can walk straight to roughly the right corner and hand you five good ones in seconds, even if a marginally better book sits two aisles away.
That is approximate nearest neighbour retrieval. Everything in the catalogue is turned into a point in a space where similar things sit near each other, and a prebuilt map lets the system look in one small corner of it instead of everywhere. Giving up on finding the absolute best match is exactly what makes answering in a fraction of a second possible.