Mind2Web
Ask a web agent to find a direct flight under a price cap and hold the seat, and it has to work in a real interface — thousands of elements, inconsistent labels, nothing laid out for a machine. Mind2Web is a benchmark built to ask exactly that kind of question.
What it contains
Released in 2023, it collects over 2,000 open-ended tasks written by crowdworkers across 137 real websites in 31 domains, each paired with the action sequence a person used to complete it on a captured page snapshot. Scoring is offline: the agent's chosen action is replayed against the saved page and compared with the annotation, on
- element accuracy — did it target the right element on the page;
- operation F1 — was the action, and any value typed, correct;
- step success rate and task success rate, the latter demanding every step be right.
Why it matters
Its three splits are the point: cross-task, cross-website and cross-domain, separating an agent that has learned one site from one that can handle a site it has never seen. It also made a practical problem unmissable — the raw HTML of a real page dwarfs a context window — which pushed the field towards a two-stage design where a small ranker shrinks the page before the model chooses. Offline replay buys reproducibility at a price: a different but perfectly valid path scores as failure, which is why interactive suites like WebArena and OSWorld sit alongside it rather than replacing it.
Mind2Web is a benchmark for generalist web agents: over 2,000 crowdsourced tasks on 137 real websites across 31 domains, each with an annotated human action sequence. Agents are scored offline against saved page snapshots on element accuracy, operation correctness, and step and task success. Its cross-task, cross-website and cross-domain splits are the point — they separate an agent that has memorised one site from one that generalises to sites it has never seen.
A driving test tells you far more than a lap around an empty car park. Mind2Web is the driving test for AI that uses websites.
Researchers collected more than two thousand everyday jobs — book this, order that, unsubscribe from the other — spread across 137 real websites, and recorded the clicks and typing a person actually used to finish each one. An AI is then asked to do the same jobs, and marked on whether it picked the same buttons and filled in the same boxes.
The clever part is that some of the sites it is tested on are sites it never practised on, which shows whether it has learned how the web works or just memorised a few pages.