Tau-bench
Most agent benchmarks hand over a task and wait for an answer. Tau-bench does something closer to real work: it drops the agent into a customer-service conversation where the person on the other end is itself a language model, the company has a written policy, and the only way to finish is to call the right tools in the right order.
What a run looks like
Built by Sierra and covering retail and airline support domains, each task gives the agent four things to juggle:
- A simulated user who knows what they want but not how to ask for it, and who reveals details only when asked;
- A tool set over a live database — look up an order, change a booking, issue a refund;
- A policy document the agent must obey, including rules that forbid what the user is asking for;
- A goal state — success is the final database matching what a correct run would have produced, not the wording of the reply.
Why it matters
Tau-bench measures the two things production support needs and most benchmarks miss: pulling missing information out of a conversation, and declining when the rules say no. It also reports pass^k, the share of tasks solved on every one of k attempts, which falls away far faster than a single-run average. That gap is the finding, not a footnote — an agent that succeeds most of the time is not an agent you hand a refund button to.
Tau-bench is a benchmark for tool-using agents in customer-service settings, built by Sierra. Each task pairs the agent with a language-model-simulated user, a database it manipulates through tool calls, and a written domain policy it must follow; success is judged by the final database state. It also reports pass^k, the share of tasks solved on every one of k attempts, which exposes how inconsistent agents remain across repeated runs.
Imagine testing a new call-centre hire with a role-play. An actor phones in with a vague complaint, there is a rulebook of things the company does not allow, and afterwards you check the account records to see whether the right thing was actually done.
Tau-bench is that role-play, run on AI agents. The caller is played by another AI, the agent has to ask questions to work out what is really wanted, and it has to say no when the rules say no. It also scores whether the agent gets it right every time, not just usually — because a refund button that works most days is not good enough.