WebArena
Claiming an agent can use the web is easy; proving it is not, because the live web changes under you and most of it cannot be clicked safely in a test. WebArena solves that by building a web of its own.
What is actually in it
WebArena is a benchmark environment of self-hosted, fully functional websites — an e-commerce store, a Reddit-style forum, a GitLab instance and a content-management system, plus a map and a wiki for reference — paired with a set of tasks written in plain language, such as cancelling the most recent order or finding the cheapest listing and posting it to the forum. The agent drives real pages in a real browser. Scoring is functional, not textual: the harness inspects the resulting system state to check the task was actually completed, so a confident write-up of work never done scores zero.
Why that design matters
Three properties make it a benchmark rather than a demo:
- Reproducible — the sites ship as containers, so the same task means the same thing months later.
- End-to-end — one wrong click eight actions in sinks the whole trajectory, which is how real failures work.
- Real side effects, contained — the agent genuinely places the order, something a static page scrape cannot test.
What the scores said
Humans finish close to 78% of the task set; the first GPT-4-based agents cleared roughly 14%. That gap turned long-horizon web competence from an anecdote into a number. WebArena now sits alongside Mind2Web and OSWorld in GUI-agent evaluation, and low scores trace to the same causes: state lost between steps, no recovery from a failed action, and no way to tell a finished task from an abandoned one.
WebArena is a benchmark for web-browsing agents: a set of self-hosted, functional websites — shop, forum, code host, CMS — plus natural-language tasks whose completion is scored by inspecting the resulting system state rather than the agent's text. It matters because it measures long-horizon browser competence reproducibly and end-to-end; early agents scored far below human performance on it, locating the weakness of autonomy in execution rather than language.
Imagine testing a new employee by giving them a working copy of your company's websites and a list of errands: cancel this order, find the cheapest item, post this notice. You don't grade their explanation — you check whether the order was actually cancelled.
WebArena is that test, built for AI. It runs its own shop, forum and code site, hands the software a job written in plain English, then looks at the sites afterwards to see whether the job got done. Early results were humbling: people finished about four errands in five, the software barely one in seven.