BrowseComp
Most benchmarks get easier as models get better at writing answers. BrowseComp was built to resist that: it asks questions whose answers are genuinely hard to find on the web but trivial to check once you have them.
How the questions are built
Construction runs backwards from a known answer. A writer starts with an obscure fact, then composes a question with several interlocking constraints — a person, a date range, an unusual detail — such that:
- No single search returns it; the answer sits several hops from any obvious query.
- A model cannot recall it, and a short browsing session will not stumble on it.
- The answer is short and unambiguous, so grading needs no judgement call.
Released by OpenAI, the suite runs to over a thousand such problems, each verified to be unfindable by a quick lookup.
What it actually measures
The skill under test is persistence, not knowledge. Scoring well means forming a search strategy, recognising a dead end early, backtracking, reformulating a query several times, and holding partial constraints across dozens of page visits without losing the thread. That is why BrowseComp separates real browsing agents so sharply from a model with a search tool attached: accuracy climbs with the number of steps an agent is allowed and with how well it manages its own context. Its blind spot mirrors its design — real research questions rarely have one short verifiable answer, so the benchmark rewards needle-finding over synthesis and judgement.
BrowseComp is a benchmark for browsing agents, published by OpenAI, made of short-answer questions whose answers are hard to locate on the web but easy to verify. Each one demands many search steps, backtracking and reformulation rather than recall. It measures an agent's persistence and search strategy, and separates a genuine browsing loop from a model with a search tool bolted on.
Some questions are hard not because they are complicated, but because the answer is buried. Think of a quiz question like "which film did that actor's first drama teacher appear in?" — you would have to follow a trail of several clues, and you would know the answer when you found it.
BrowseComp is a test made of questions like that, used to judge AI systems that browse the web for you. Each answer is short and easy to check, but finding it takes many searches, dead ends and fresh attempts.
It measures stubbornness more than cleverness: whether the software keeps going sensibly instead of giving up or guessing.