OSWorld 2.1 is the current release of OSWorld 2.0, a benchmark from XLANG Lab that tests computer-use agents on 108 long-horizon workflows across everyday and professional work. Snorkel AI served as research and data partner and supports the work through its Open Benchmarks Grants program. At release, the best agent completed only about one task in five, and on the OSWorld 2.0 leaderboard, Claude Opus 5 now reaches 44.3%.
What is OSWorld 2.1?
OSWorld 2.0 is a benchmark of 108 long-horizon computer-use workflows. Each one is a realistic end-to-end job that takes a human a median of about 1.6 hours, roughly 48× longer than a typical OSWorld 1.0 task. Agents work inside real operating-system environments, across desktop applications and 31 self-hosted web services.
OSWorld 2.1, released on September 16, 2026, is the recommended version for running the benchmark. It is a bug-fix release with updated agents, evaluation workflows, and pinned references for the task files, assets, mocked websites, and provider images, so results from different teams can be compared on the same footing. Track live scores on the Snorkel AI OSWorld 2.0 leaderboard.
108
Long-horizon computer-use workflows
~1.6 hrs
Median human completion time
318
Avg. tool calls per task (Claude Opus 4.7, max)
20.6%
Best binary completion at OSWorld 2.0 release (Opus 4.8)
Why OSWorld 2.1 matters
Agents now score above 80% on the original OSWorld, which asked for short tasks in mostly one application at a time. Real work looks different. It spans several tools, depends on information that shows up halfway through, and needs the agent to check its own output.
OSWorld 2.0 closes that gap. Tasks are built from authentic input artifacts, such as real emails and scanned receipts, and cross-referenced against stateful user profile data. Each task also ships with a separate safety report that audits safety-sensitive execution.
| Measure | OSWorld 1.0 | OSWorld 2.0 / 2.1 |
|---|---|---|
| Task horizon (agent steps) | Under 30 | Over 250 |
| Cross-app tasks | A minority | The majority (2+ apps or services) |
| Self-hosted web environments | None | 31 websites |
| Input artifacts | Mixed or synthetic | Authentic |
| Annotated challenge tags | None | 10 |
What the tasks stress
The benchmark targets challenge phenomena that are common in real workflows and thin in earlier benchmarks.
- Interaction design: streaming interaction and dynamic environments, where the screen changes while the agent is acting.
- Agent patterns: cross-source reasoning, implicit-state inference, and visual-spatial precision.
Tasks are graded against many weighted checkpoints, about 27 per task on average. The primary metric is binary completion, where every checkpoint must pass. A partial score reports the average share of checkpoints met.
Key findings
Under the primary binary-completion metric at 500 steps:
- At release, no agent was close. Claude Opus 4.8 with maximum thinking and batched tool calls led with 20.6% completion and a 54.8% partial score. Since then, Claude Opus 5 has reached 44.3% binary and 77.7% partial on the OSWorld 2.0 leaderboard.
- Efficiency and accuracy diverge. GPT-5.5 is far more token-efficient but plateaus near 13%.
- Failures are not about clicking. Agents lose track of constraints, miss information that arrives mid-task, guess instead of asking the user, and skip verification. They struggle most when a task depends on hidden state they have to recover.
| Model and configuration | Binary completion | Partial score |
|---|---|---|
| Claude Opus 4.8 (max thinking, batched tools) | 20.6% | 54.8% |
| Claude Opus 4.7 (max thinking, batched tools) | 18.2% | 48.9% |
| Claude Opus 4.7 (max thinking) | 13.9% | 49.1% |
| GPT-5.5 | ~13% | — |
| Claude Sonnet 4.6 (medium thinking) | 9.3% | 33.9% |
| Claude Sonnet 4.6 (max thinking) | 8.3% | 41.5% |
| MiniMax M3 | 4.6% | 22.3% |
| Kimi 2.6 | 4.6% | 22.1% |
| Qwen 3.7-Plus | 2.8% | 21.5% |
Results at a 500-step budget. The GPT-5.5 figure is the paper’s approximate value. Check the official leaderboard for current numbers.
For teams building computer-use agents, the takeaway is concrete: at release, even the strongest agent finished about one realistic workflow in five, and the best score on the OSWorld 2.0 leaderboard is still only 44.3%. The gap comes from tracking constraints, catching mid-task changes, and verifying work, not from basic GUI control.
Running OSWorld 2.1
The harness supports Docker, for Linux servers with KVM, and AWS, for large parallel runs. Task classes and full assets are distributed through gated Hugging Face datasets, which keeps answers and evaluator details out of reach of agents being evaluated. Access is approved automatically after you accept the terms. Mocked websites are self-hosted, and GitLab-backed tasks need your own GitLab instance and token.
To have results verified and shown on the public leaderboard, you schedule a run with the maintainers and share your agent implementation under the OSWorld framework. See the repository for the public evaluation guide.
Authors
OSWorld 2.0 was created by Mengqi Yuan, Zilong Zhou, Xinzhuang Xiong, and 33 co-authors from XLANG Lab and collaborating institutions, 36 authors in all. Snorkel AI’s Zhengyang Qi, Vincent Sunn Chen, and Frederic Sala contributed to the work. The paper was submitted to arXiv on June 28, 2026.
Snorkel AI’s role
Snorkel AI is the research and data partner for OSWorld 2.0, with researchers Zhengyang Qi, Vincent Sunn Chen, and Frederic Sala among the co-authors. Snorkel also supports OSWorld through its Open Benchmarks Grants program, which backs open benchmarks, red-teaming, and fellowships for open AI evaluation. See every funded benchmark. The work reflects Snorkel’s broader focus on rigorous, expert-grounded evaluation of AI agents on realistic work. It builds on Snorkel’s data-centric AI research.
OSWorld 2.1 is part of Snorkel’s broader work on AI agent evaluation. Explore related benchmarks: Agents’ Last Exam, Terminal-Bench Science, and Terminal-Bench 2.0.

