Notes
🔍

Token Budget (Agent)

An agent's cost does not look like a chatbot's. Every iteration re-sends the whole accumulated transcript — the plan, every tool call, every observation — so the tenth step is far more expensive than the first, and a run that loops a few extra times can cost several times what you expected. A token budget is the ceiling that makes that bounded.

Where the budget is applied
Runtimes enforce it at more than one level:

  • Per run — a total allowance across all iterations, counted as the loop goes;
  • Per step — a cap on a single model call, so one enormous tool result cannot swallow the lot;
  • Per sub-agent — each delegated worker gets its own slice and cannot drain the parent's allowance;
  • A reserve — tokens held back so the agent can still write its answer once the work is done.

What happens as the budget runs low is the real design decision. Compacting the context buys more steps at the cost of detail; summarising and handing over preserves the work; terminating is at least honest. Doing nothing means the run dies mid-task with nothing to show for it.

Why it matters
A token budget turns an open-ended reasoning loop into something with a price you can quote in advance. It also makes failure legible: a run that stopped at its ceiling tells you either that the task needed more room or that the agent was going in circles, and both are things you learn long before the monthly invoice arrives.

A token budget is a cap on the tokens an agent may consume — per run, per model call, or per delegated sub-agent — enforced as the loop iterates. It exists because each iteration re-sends the whole accumulated transcript, so cost climbs sharply with the number of steps. The budget bounds that, and the policy attached to it (compact the context, summarise and hand over, or terminate) decides whether exhaustion is a managed outcome or a run lost mid-task.

Think of a taxi meter rather than a bus fare. You are not paying a fixed price for one answer — you are paying for every minute the AI keeps working, and an agent that takes a long route costs a great deal more than one that goes straight there.

A token budget is the limit you set before the meter starts. Tokens are the units AI models are billed in, roughly small chunks of text. Setting a budget says: get as far as you can on this much, then stop. It keeps a curious experiment from turning into a startling bill, and it forces you to decide in advance what the agent should do when the money runs out.