Definition
Computer use is an AI capability enabling agents to perceive graphical user interfaces and take actions — clicks, keystrokes, scrolling — across browser, mobile, and desktop environments. Models generate UI action function calls that client-side code executes.
Key Points
-
2026-07-29: polar-browser positions Chromium-fork agentic-browser for multi-hour logged-in knowledge work; $5.7M seed (2026-08-02-techcrunch-polar-5-7m-seed)
-
2026-07-24: prentis fundraising talks highlight office-workflow computer-use race; claude-opus-5 claims OSWorld gains (2026-07-24-prentis-100m-raise-talks-computer-use, 2026-07-24-anthropic-claude-opus-5-launch)
-
2026-07-01: github-copilot browser tools GA in VS Code — integrated browser without external MCP; tab isolation, enterprise domain controls (2026-07-01-github-copilot-browser-tools-ga)
-
2026-06-24: google integrated computer use as built-in tool in gemini 3.5 Flash (previously standalone Gemini 2.5 model) (2026-06-24-google-gemini-35-flash-computer-use)
-
Access: Gemini API and Gemini Enterprise Agent Platform
-
Environments: Browser (default), mobile (
ENVIRONMENT_MOBILE), desktop (ENVIRONMENT_DESKTOP) -
Safety: Adversarial training, optional user confirmation for sensitive actions, automatic stop on indirect prompt injection detection
-
Limitations: Struggles with CAPTCHAs, dynamic content, unexpected layouts; not mature for unsupervised operation
Prompt injection risks remain despite automatic stop feature — security limitations should be noted in deployments.
Related
- hive-32b
- computer-use-agents-race
- github-copilot
- gemini
- agentic-ai
- tool-use
- prompt-injection
- automation