Computer-Use Agent
Most agent tools are APIs: clean, documented, built to be called. A computer-use agent gives all that up and works the way you do — it looks at a screenshot of a desktop, decides where to click, and types. The point is reach: software with no API, a legacy internal tool, a PDF viewer, anything on screen becomes operable.
The loop in pixels
Each turn the agent receives an image of the screen and emits a low-level action — click at a coordinate, type a string, scroll, press a key combination, take a fresh screenshot. Nothing is semantic; there is no element tree to query, so the model has to ground its intent in the image, locating the right button by appearance and position. Anthropic Computer Use and comparable offerings expose exactly this loop, and OSWorld measures it on real desktop applications across multi-step jobs.
Why it is hard
The generality is paid for three times over:
- A mis-grounded click is silent. The agent clicks the wrong thing, the screen changes unexpectedly, and the next screenshot has to be read from scratch.
- The loop is slow and expensive: one screenshot per step, dozens of steps for work a script would finish in one call.
- Anything rendered on screen is untrusted input. Text in a web page or a document can carry a prompt injection aimed at the agent reading it, which is why these agents run on a sandboxed virtual desktop with narrow credentials rather than on the user’s own machine.
A computer-use agent operates a graphical desktop directly: it receives screenshots, then emits clicks, keystrokes and scrolls as its actions. Because it drives the same interface a person does, it can work software that exposes no API at all. The cost is a slow, fragile loop — targets must be located in pixels, wrong clicks fail silently, and on-screen text is untrusted input — so it runs sandboxed. OSWorld benchmarks the capability.
Imagine a new colleague nobody can give special access to, so you sit them at an ordinary computer with a mouse and keyboard. Slower than a shortcut — but they can use any program on the machine.
A computer-use agent works that way. It takes a picture of the screen, decides where to click, clicks, then looks again. Nobody has to build it a special connection to the software first.
That generality is the appeal, and the clumsiness is the catch: it misreads buttons, it is slow, and anything written on the screen can try to mislead it. So it usually works on a separate, locked-down machine.