Retry & Backoff (Agent)
A tool call fails for two very different reasons: because it was wrong, or because the world was briefly busy. A rate limit, a timed-out request, a database mid-failover — none of these mean the agent made a mistake, and all of them are cured by waiting a moment and asking again. Retry and backoff is the policy that decides when to wait, how long, and when to stop.
The pieces of the policy
They are small and they interlock:
- Classification: retry transient failures — 429, 503, connection resets — and never retry a malformed request or a schema violation, which will fail identically forever.
- Exponential backoff: wait one second, then two, then four, so a recovering service is not hammered by the client depending on it.
- Jitter: a random offset on each delay, so parallel agents do not all come back at the same instant and recreate the overload.
- A ceiling: a fixed attempt count, after which the failure is handed to the loop as an observation the model must reason about.
Why an agent needs it specifically
A single model call either returns or does not. An agent makes dozens of calls across a long run, so a one-percent transient failure rate becomes near-certain failure somewhere in the trace. The opposite extreme is the real danger: without classification and without a ceiling, retrying becomes an unproductive loop that drains the token budget re-sending a request that was broken to begin with. That is why retry policy sits beside step limits and budgets as a runtime control, rather than being buried inside each tool.
Retry and backoff is the policy an agent applies to failed tool calls: re-issue those that failed transiently — rate limits, timeouts, connection resets — after an exponentially growing, jittered delay, up to a fixed attempt ceiling, and never re-issue a malformed request. It matters because a long run makes many calls, so transient failures are near-certain; without a ceiling, retrying becomes an unproductive loop that drains the token budget.
When a phone call doesn't connect, you try again — and if it still fails, you wait a bit longer before the next attempt instead of redialling nonstop. Retry and backoff is that same habit written into software. When a program asks another service for something and gets no useful reply, it waits a moment and asks again, waiting longer each time.
The two important rules are knowing when not to bother — if the request itself was wrong, repeating it will never help — and knowing when to give up. Without a limit, a program can spend hours and real money asking the same question over and over.