Notes
🔍

KV Cache Budget

When an on-device language model “remembers” the earlier parts of a conversation while generating the next word, that memory is not free. A KV cache budget is the portion of device memory deliberately reserved for that short-term memory, so the model can respond quickly without running out of RAM.

What the cache stores

For every token the model reads or generates, each transformer layer creates key and value tensors. These are saved in the KV cache. At the next token, the model reuses them rather than recomputing attention for the entire prior prompt. This makes token-by-token generation practical, but the cache grows linearly with context length. Its approximate size depends on:

  • the number of transformer layers;
  • the number and dimension of KV attention heads;
  • the maximum number of retained tokens;
  • the storage precision, such as FP16, FP8, or quantized cache values; and
  • the number of simultaneous conversations or requests.
Why it is a deployment budget

Model weights and the KV cache compete for the same finite memory. A phone might fit a 4-bit quantized model comfortably at first, then fail after a long prompt because its cache has consumed the remaining RAM. A KV cache budget therefore sets a hard operating limit: for example, reserve 1 GB for cached tokens, cap context at 4,000 tokens, and leave memory for the runtime, operating system, and temporary activations. Runtimes such as llama.cpp, TensorFlow Lite–based LLM stacks, and mobile inference engines use this planning to avoid crashes, swapping, and severe slowdowns.

Design trade-offs

A larger budget supports longer documents, richer chat history, and multiple users. A smaller one lowers memory pressure and can reduce energy use, but forces truncation of older context. Systems handle this with sliding-window attention, prompt summarization, fewer concurrent sessions, or KV-cache quantization. On a thermally constrained handset, getting this budget right is the difference between a private offline assistant that remains responsive and one that stalls halfway through a conversation.

KV cache budget is the amount of device memory reserved for an LLM’s stored attention keys and values from prior tokens during generation. It sets the maximum practical context length, concurrent sessions, and batch size after model weights and runtime overhead are accounted for. On-device, this budget is critical: an oversized cache causes memory pressure or failure, while a tight budget requires context truncation, cache quantization, or eviction.

Think of a language model’s KV cache budget as the amount of desk space it sets aside for notes about a conversation. As you chat, the model needs to keep useful notes from earlier words so it can answer in context rather than treating every message as brand new.

On a phone or other small device, memory is limited. The cache budget decides how much conversation history the model can comfortably hold. A larger budget allows longer chats or longer documents, but uses more of the device’s memory. A smaller one saves space, but the model may eventually lose track of earlier details.