Memory Retrieval (Agent)
Storing everything an agent has ever learned is the easy half. The hard part arrives at step forty, when it needs exactly one fact from that pile and has room in the prompt for almost nothing else. Memory retrieval is the act of choosing what comes back.
How the choice is made
Retrieval runs against the memory store just before the model is called, and most systems blend several signals:
- Semantic similarity: the current task is embedded and compared against embedded memories in a vector store, so a note about the staging database surfaces for a question about logging in.
- Metadata filters: this user, this project, this session — narrowing the pool before similarity ranks what is left.
- Recency and importance: weights that keep a fresh or flagged memory ahead of an older, vaguely similar one.
Whatever survives is cut to a token budget and pasted into the prompt as ordinary text. The model never queries the store itself unless retrieval is handed to it as a tool.
Why it matters
Retrieval is what makes a long-term store behave like memory instead of an archive, and its failures show up directly in behaviour. Retrieve too little and the agent redoes work it finished an hour ago, or asks for something it was already told. Retrieve too much and the recalled text crowds out the instructions and the plan, and the run drifts. Retrieve something stale and it acts confidently on a fact that stopped being true. LangGraph's store API and LlamaIndex's retrievers are where this is usually wired up.
Memory retrieval is the step that decides which stored memories enter an agent's prompt for the current step — usually embedding similarity over a vector store, narrowed by metadata filters, weighted for recency, then cut to a token budget. It is what turns a growing memory into usable context: retrieve too little and the agent repeats finished work, too much and the recalled text crowds out the task it was given.
Imagine an assistant with a filing cabinet holding everything you have ever told them. The cabinet is only useful if, when you ask about your trip to Lisbon, they come back with the two pages about Lisbon — not the whole drawer.
Memory retrieval is that fetching step for an AI assistant. It keeps notes from earlier work, and before each new step it picks out the few notes that look relevant and puts them in front of itself to read.
Get it right and the assistant remembers what you told it last week. Get it wrong and it either forgets, or buries the actual task under a pile of old notes.