GAIA Benchmark
Most benchmarks that models find hard, people find hard too. GAIA inverts that. Its questions are ones a capable adult could answer with a browser and half an hour — and that a bare language model cannot answer at all, because the answer is not in its weights. It has to be assembled.
What the questions look like
The GAIA benchmark — General AI Assistants, published in late 2023 by researchers at Meta AI and Hugging Face — holds a few hundred questions with short, unambiguous answers graded by exact match rather than by a judge. Getting one right means doing real work:
- finding and reading several web pages, not just one;
- opening an attached file — a spreadsheet, a PDF, an image — and pulling a figure out of it;
- chaining results, where step four depends on what step two returned;
- arriving at one specific answer: a name, a date, a number.
Why the score is informative
The gap GAIA opened was stark: human annotators answered around 92% correctly, while a strong model with browsing and plugins managed roughly 15%. That spread is the whole point. The reasoning each question needs is unremarkable, so the difficulty lives entirely in the agent layer — planning, tool selection, and keeping track of findings across steps. Grading stays cheap and honest because the answers are exact strings, which also makes the questions tiered: levels one to three reflect how many steps and tools a question demands. Read GAIA as the general-assistant counterpart to SWE-bench: one asks whether an agent can see a varied real-world errand through, the other whether it can ship a code change.
GAIA (General AI Assistants) is a benchmark of a few hundred real-world questions that require multi-step web research, file handling and tool use, each with one short answer graded by exact match. Humans score around 92%, early tool-equipped language models around 15%. Because the reasoning per question is ordinary, GAIA measures the agent layer itself — planning, tool selection and carrying state across steps — rather than raw model knowledge.
Imagine a test made of errands rather than exam questions. Find out which of these three museums has the oldest item in its collection, then tell me the year. Nothing about it is clever — but you would need to search, read a few pages, open a spreadsheet, and keep track of what you found.
GAIA is a set of a few hundred such tasks, used to check whether AI systems can actually run errands like these. People get around nine in ten right. AI assistants, when the test first appeared, managed closer to one in seven. The gap is useful: it shows the hard part is not knowing things, it is seeing a job through from start to finish.