OSWorld
A benchmark made of multiple-choice questions cannot tell you whether an agent can rename a column in a spreadsheet and email the file. OSWorld can, because it hands the agent a real computer and then inspects the computer afterwards.
What the benchmark is
Published in 2024, OSWorld is an executable environment for computer-use agents:
- A real desktop inside a virtual machine — Ubuntu, with Windows and macOS variants — not a simulated mock-up of one.
- 369 tasks drawn from genuine workflows across a browser, a terminal, an IDE, GIMP, LibreOffice and the file manager, including tasks that span several applications.
- The agent perceives screenshots and the accessibility tree, and acts by moving the mouse and typing.
- Each task ships a setup script and an execution-based validator: the grader checks the resulting file or system state, so credit comes from the outcome and not from a plausible-looking trajectory.
Why it matters
OSWorld made the gap between a demo and a capability measurable. People complete roughly three-quarters of the tasks; the first agents evaluated on it managed a little over a tenth, and the failure modes the authors catalogued — misreading a GUI, clicking a control that had not finished rendering, never recovering from one wrong click — mapped out the work that followed. It sits with WebArena and Mind2Web at the GUI-interaction end of agent evaluation, and it is the standard reference when a vendor claims a model can operate a computer.
OSWorld is a benchmark for computer-use agents that runs them on a real desktop operating system inside a virtual machine, scoring 369 tasks across a browser, terminal, IDE and office applications. Every task carries a setup script and an execution-based validator that inspects the resulting file or system state, so an agent earns credit for the outcome rather than for a convincing-looking attempt. It quantified how far GUI agents still trail human operators.
It is easy to film an AI doing something impressive once. Proving it can reliably use a computer is much harder — so OSWorld sits the AI down at a real one.
It is a test built from hundreds of ordinary chores: edit a document, change a setting, pull figures off a website into a spreadsheet. The AI sees the screen and works the mouse and keyboard, just as a person would. Afterwards the test checks the actual result — was the file really saved correctly? — rather than whether the AI sounded confident about it. People manage most of these chores; the first AI attempts managed very few, which is exactly why the test matters.