Vol. VII · Protocols & Tools · VII.4
Computer Use
The environment is a GUI or a browser, not an API. browser-use, OpenHands, and Anthropic's computer-use all shrink the web or desktop into an action space.
6 min read
Doctrine
- Prefer APIs and MCP servers over GUI when they exist. Computer use is the fallback, not the first tool.
- Action spaces should be small: click, type, scroll, screenshot — with DOM snapshots, not pixels only, when possible.
- Always isolate. A browser agent with the user's cookies is a session hijack waiting to happen.
The expensive environment
Anthropic listed computer-use as the motivating autonomous-agent example: you cannot hardcode the path through arbitrary software. That does not make it the default. browser-use on Playwright with a DOM snapshot is the current open standard. OpenHands adds a full desktop for software engineering.
Anti-patterns
- Pixel-only agents on dense enterprise UIs without an accessibility tree.
- Computer-use against banking or admin consoles without a human gate.