Why It Matters
For computer-use agents, OSWorld is a close proxy for production difficulty because the agent must ground pixels/UI state, coordinate applications, mutate files, and survive long-horizon desktop workflows.
Problem
- Many digital-agent benchmarks are static, text-only, web-only, or limited to one application/domain.
- GUI agents need to handle visual grounding, OS state, files, browser tabs, desktop apps, and cross-application workflows.
- Action matching is too brittle for open-ended computer tasks where many valid paths can reach the target state.
- Live or ad hoc desktop evaluations are hard to reproduce without controlled setup and executable validators.
Key gap: The paper asks how to build a realistic, reproducible benchmark for multimodal agents that use real computer environments rather than simplified interfaces.
Method
Create a benchmark inside controllable desktop environments, define real-world tasks with setup configuration and execution-based validators, run agents through a standardized computer-control interface, and compare success rates with human performance.
- Prepare a virtualized desktop environment and load the task-specific initial state.
- Give the agent a natural-language instruction and a multimodal computer observation.
- Let the agent execute actions against real web and desktop applications over multiple steps.
- Collect screenshots, actions, videos, files, and other artifacts during the run.
- Run the custom task evaluation script against the final state.
- Aggregate success rates across 369 tasks and compare model baselines with human performance.
- Analyze failure modes such as GUI grounding, operational knowledge, layout robustness, and history handling.
Claims
C1 · paper-supported
OSWorld introduces a scalable real computer environment for multimodal agents across operating systems including Ubuntu, Windows, and macOS.
Scope: Abstract.
- E-real-computer-env: The paper describes OSWorld as a scalable real computer environment for multimodal agents with task setup, execution-based evaluation, and interactive learning across Ubuntu, Windows, and macOS.
C2 · paper-supported
The benchmark contains 369 computer tasks involving real web apps, desktop apps, OS file I/O, and workflows across multiple applications.
Scope: Abstract and project page statistics.
- E-task-count: OSWorld contains 369 tasks involving real web and desktop apps, open domains, OS file I/O, and multi-application workflows.
C3 · paper-supported
Each task includes detailed initial-state setup and a custom execution-based evaluation script for reproducible grading.
Scope: Abstract and environment infrastructure description.
- E-setup-eval: Each task uses detailed initial state setup and custom execution-based evaluation scripts; the environment infrastructure includes task initialization, agent interaction, post-processing, retrieval, and evaluation.
C4 · paper-supported
OSWorld is designed for multimodal and cross-application computer-use tasks, including intermediate initial states and many execution-based evaluation functions.
Scope: Project page comparison table.
- E-feature-table: The project page lists OSWorld as supporting a controllable executable computer environment, multimodal support, cross-app tasks, intermediate initial state, and 134 execution-based evaluation functions.
C5 · paper-supported
The paper reports that humans complete over 72.36% of tasks while the best evaluated model achieves only 12.24% success.
Scope: Abstract.
- E-human-model-gap: Humans accomplish over 72.36% of OSWorld tasks while the best evaluated model achieves 12.24% success.
C6 · paper-supported
The authors attribute major model deficiencies to GUI grounding and operational knowledge, with further analysis of resolution, history, layout/noise robustness, and OS transfer.
Scope: Abstract and project page analysis.
- E-failure-analysis: The paper emphasizes GUI grounding and operational knowledge failures; the project page summarizes analysis on screenshot resolution, trajectory history, UI layout/noise robustness, and cross-OS performance correlation.
C7 · paper-supported
Running OSWorld is itself a systems task: the code supports local VM providers and Docker/KVM/cloud-style parallel execution, and records artifacts for result inspection.
Scope: Project repository setup and experiments instructions.
- E-runtime-requirements: The repository documents VMware/VirtualBox setup, Docker with KVM support, AWS-style parallel evaluation, headless runs, baseline agent commands, and result artifacts including screenshots/actions/videos.
Agent Infrastructure Takeaways
- Computer-use evaluation is runtime infrastructure, not just data curation: reset, isolation, credentials, files, GUI state, and validators all matter.
- Execution-based validators are the right primitive for open-ended desktop tasks because many action paths can be valid.
- Multimodal agents fail on grounding and operational procedures, so benchmark logs must preserve screenshots, actions, videos, and final artifacts.
- VM/provider choice affects reproducibility, parallelism, cost, and security; these are benchmark design variables.
- Desktop agents need durable execution and cleanup because interrupted runs can leave containers, files, accounts, or app state behind.
- OSWorld sits downstream of WebArena and tau-bench: it expands from websites and business APIs to full operating-system control.
Caveats
- The benchmark is much heavier to run than text-only or browser-only evaluations.
- Some tasks involve credentials or network-sensitive services, and the project notes Google Drive setup issues for 8 tasks.
- The reported 12.24% best-model score is historical and not the current leaderboard ceiling.
- VM images, provider support, and setup scripts are part of the benchmark surface and can drift over time.
- Execution-based evaluation is only as good as the task validator and setup isolation.
- Full-computer tasks are faithful but can make failure attribution harder because perception, planning, app knowledge, and infrastructure issues mix together.