AgentBench
Most language-model benchmarks ask a question and mark the answer. That tells you very little about whether a model can hold a plan together across twenty interactions with a system that pushes back. AgentBench was among the first suites built to measure the second thing instead of the first.
What it tests
AgentBench drops a model into eight distinct interactive environments and scores it on completing multi-turn tasks in each — among them a Linux shell, a SQL database, a knowledge graph, a card game, a simulated household, and web shopping and browsing. Every task is open-ended and stateful: the model issues a command, receives the environment's real output, and must decide what to do next. That design surfaces failures a single-turn benchmark cannot see:
- Losing an instruction that was given fifteen steps earlier.
- Emitting a malformed command and then repeating it unchanged.
- Declaring success without checking that the environment actually changed.
Why it matters
Its headline finding — a wide gap between leading commercial models and open ones that scored comparably on conventional benchmarks — established the point the field now takes for granted: reasoning quality on a single prompt does not predict competence inside a loop. AgentBench is a broad capability probe, so it sits alongside narrower suites rather than replacing them. Reach for SWE-bench on code, GAIA on research-style tool use, WebArena on browser work. Reported together, they describe an agent's shape far better than any single number does.
AgentBench is a benchmark suite that measures a language model's ability to act as an agent across several interactive environments — operating systems, databases, knowledge graphs, web browsing, games — scoring task completion over multi-turn interaction. It tests tool use and sustained decision-making rather than single-response quality. Results give a comparable capability baseline across models, which text-only benchmarks cannot supply for agentic deployment.
You can test someone's cooking by asking them to name the ingredients in a recipe. Or you can put them in a kitchen and ask for dinner. The second test is much harder, and much more honest.
AgentBench is the kitchen version for AI. Instead of asking questions, it drops the software into eight different working environments — a computer terminal, a database, an online shop, a simulated house — and gives it real jobs that take many steps to finish. It made one thing clear: software that answers quiz questions brilliantly can still be hopeless at actually getting a job done.