OSWorld 2.0: Why Long-Horizon Computer-Use Agents Still Fail Four Out of Five Tasks

At our latest Snorkel AI Reading Group, held virtually for the first time, Mengqi Yuan (XLANG Lab, University of Hong Kong) presented OSWorld 2.0, a benchmark of 108 long-horizon, real-world computer-use workflows spanning 31 self-hosted websites and professional desktop applications. Tasks average more than 300 agent steps, 69.6% take a skilled human over an hour to finish, and the best system in the paper still completes only 20.6% of them outright. OSWorld 2.0 was developed at XLANG Lab with support from Snorkel AI’s Open Benchmarks Grants program, and it was one of the benchmarks Anthropic used to evaluate Claude Fable 5.1 when that model launched less than a day before this talk. The question Yuan’s team set out to answer: once a benchmark stops looking like a short, clean scripted task and starts looking like an actual afternoon of work, how far do today’s frontier agents actually get?

Transcript

Lightly edited for readability.

Why OSWorld 2.0: OSWorld 1.0 was saturating

Yuan opened by explaining the motivation. OSWorld 1.0, the team’s original benchmark, evaluates computer-use agents on real-world tasks. It had done its job: agent scores climbed for two years, and by earlier this year the best systems were clearing 80% accuracy. “It’s nearly saturated,” Yuan said, “but we all know that in real-world tasks, computer agents are not that perfect.” A benchmark that’s been solved stops telling you anything useful about where agents actually break, so the team built a harder, messier successor.

OSWorld 2.0 tests the most recent models, including Claude Opus, at a very different scale: agent trajectories now run 300 to 500 steps, and tasks routinely span multiple applications rather than one. “Maybe we have five apps together in one task,” Yuan said of the most complex cases.

Inside a task: the multi-app reimbursement workflow

To make that concrete, Yuan walked through a representative task: submitting an expense reimbursement. The agent isn’t told which apps to use or how to navigate them, it has to follow a tutorial the way a human employee would, then work across Gmail, local files, a booking app, and a bank account to assemble everything the claim needs.

The task is deliberately noisy in the way real work is noisy. Looking up an Airbnb receipt for reimbursement, the agent finds several dollar figures on the same screenshot and has to determine which one is actually correct. Some of the information it needs isn’t visible at all until it checks prior submission history in the reimbursement system, where an earlier record reveals the person ID it needs. A pop-up from Gmail interrupts mid-task with new information: a specific number has changed, and the agent has to notice that and update its plan accordingly.

OSWorld 2.0 also changes the rules around asking for help. In OSWorld 1.0, agents were told not to ask questions and simply do their best with what they had. In 2.0, the team built a simulated user the agent can query for clarification or missing information, closer to how a new hire would actually behave on the job.

What’s new: scale, self-hosting, and partial credit

Several structural changes separate OSWorld 2.0 from its predecessor. The median human time estimate per task is now about one and a half hours, up sharply from OSWorld 1.0, and the average trajectory runs over 300 steps.

The team also self-hosts the entire environment. Real websites are messy and change underneath you, which makes evaluation unstable, so OSWorld 2.0 self-hosts 31 commonly used websites instead of pointing agents at the live internet. Each task carries an average of 27 scoring checkpoints, giving partial credit rather than a single pass/fail signal (more on why that matters in the Q&A below).

To keep tasks from becoming too open-ended, the team also uses a language model as part of evaluation, but deliberately limits how much weight it carries: LLM-based judging accounts for less than 50% of scoring, because Yuan’s team found it isn’t reliable enough on its own, especially for judging visual or “virtual” information on screen.

Ten challenge phenomena, from cross-source reasoning to implicit state

The heart of the talk was a tour through the ten challenge phenomena the team built into OSWorld 2.0, each targeting a pattern where agents commonly struggle.

Cross-source reasoning requires pulling information from multiple places and synthesizing it before acting: a number from a Gmail receipt, a payment detail from a bank account, a contact from prior correspondence. “For previous benchmarks, most of the benchmarks are not that messy,” Yuan said, “but the messy road is what actually the agent needs to work on in real time.”

Virtual-special tasks cover applications like CAD software or Blender that demand a lot of specialized, non-text understanding to operate at all.

Implicit-state inference removes the safety net of being told which app to use. In OSWorld 1.0, every task named its app. In 2.0, some tasks, like completing a piece of scientific work, give the agent no such hint; it has to figure out which tool is even appropriate.

Conflicting information tasks intentionally pit two valid-looking signals against each other. In one example, an agent is asked to buy two concert tickets in adjacent seats. Take a screenshot of the seating chart and the seats look adjacent. Read the underlying HTML instead, and the two seat IDs are both listed as available but aren’t actually next to each other. The agent has to decide which representation of reality to trust.

Tutorial-following tasks come with PDF, blog, or video instructions the agent needs to actually follow rather than improvise around.

Dynamic environments change while the agent is working: a Gmail notification pops up mid-task and the agent has to notice and replan.

Streaming tasks push on a structural limitation of current agents, which operate in a “screenshot, think, act” loop. If a pop-up is moving around the screen, an agent that only checks state once between actions can be looking at stale information by the time it acts.

Proactive interaction rewards agents that notice something is off, like a personal ID for the wrong country, and ask the simulated user for correct information rather than guessing or ignoring it.

Building and verifying the benchmark

Task design pulled from four sources: interviews with domain experts, questionnaires used to seed initial task ideas, brainstorming from the team’s own daily work, and material harvested from YouTube tutorials and blog posts about what people actually do on their computers. Annotators then iterated with domain experts to make each task more complex and realistic.

Every finished task went through a verification pass: another agent attempted it and its trajectory was rolled out and reviewed, and a human cross-checked the result to confirm the task and its scoring were correct.

How frontier agents perform: Claude vs. GPT

On results, Yuan highlighted a split between accuracy and efficiency (the full breakdown, kept current as new models are added, lives on Snorkel’s OSWorld 2.0 leaderboard). Claude Opus scores higher on accuracy, but GPT models are more token-efficient, using fewer output tokens to get to their (lower) results, which matters directly for cost and generation time. Unsurprisingly, accuracy drops as a task’s estimated human completion time grows, longer tasks are harder for agents in a fairly direct, visible way.

The most striking number of the day came from outside the paper itself: Claude Fable 5.1, which Anthropic had released less than 24 hours before the talk, pushed OSWorld 2.0 partial-credit scores above 60% and binary completion above 45%, well ahead of the 20.6% binary / 54.8% partial score the paper reports for Claude Opus 4.8, the strongest model evaluated when it was written. Benchmarks age fast at the frontier.

Domain-level results showed a further split: Claude tends to do better on agent-heavy, challenging categories like dynamic environments and cross-source reasoning, while GPT models do comparatively better on virtual-specific tasks and multimodal editing.

Four failure modes

Yuan grouped the recurring ways agents fail into four buckets:

Information missing. OSWorld 2.0 tasks are dense with instructions and environment detail, and agents frequently miss pieces of it. Compounding this, agents are reluctant to ask for verification when they hit ambiguous or conflicting information, they tend to guess or ignore it rather than check.

Perception. Agents struggle to parse complex tutorials, particularly video tutorials, and often aren’t familiar enough with a given application to know how to operate it, which shows up as difficulty planning and interacting correctly.

Verification. This failure mode has two parts. First, agents often submit a task as complete when it isn’t, missing information without realizing it. Second, even when something looks off mid-task, agents are reluctant to go back and check earlier steps for mistakes.

Memory. As trajectories stretch from roughly 30 steps in OSWorld 1.0 to as many as 300 in OSWorld 2.0, agents lose track of information gathered early in the task by the time they need it later, “maybe they just read a number in the very front steps, and when the memory goes to maybe step 300, they forgot,” as Yuan put it.

Safety checkpoints

OSWorld 2.0 also scores agents on safety, not just task completion. One category checks whether an agent will avoid leaking sensitive information; a task that asks an agent to push code to GitLab includes an API key it should never actually commit. A second category penalizes side effects: successfully finishing a task by deleting a user’s files along the way is not, in fact, success. A third checks whether agents will avoid engaging with unsafe content, the team built lookalike websites similar to ones used elsewhere in the benchmark, and agents that click through to them without noticing anything is wrong take a penalty on that safety checkpoint.

Q&A

Q: How do you go about designing what partial credit means, and how you measure it?

Snorkel’s Francesca Vera opened the Q&A by asking about one of the most meaningful shifts from 1.0 to 2.0: moving from binary pass/fail to partial credit through checkpoints. Yuan explained that because OSWorld 2.0 tasks are so long, failing the whole task doesn’t mean an agent did nothing, so the team wanted a way to measure how much of a task it actually completed. Each task is broken into scored checkpoints, weighted by how important and how difficult that piece of information is to the overall task. Correctly entering a name might earn a small amount of credit; correctly handling the piece of information the whole task hinges on earns much more.

Q: Do you ever see a model pass a checkpoint that’s really complex or difficult, but trip up on something simpler?

Vera asked whether models ever nail a genuinely hard checkpoint but stumble on something simple. Yuan said this is common, and traced it back to memory rather than difficulty: frontier agents are generally good at identifying which part of a task matters most and focusing there, but they can still lose track of a number read early in a long trajectory by the time step 300 arrives, and get it wrong not because the step is hard but because the memory of it decayed.

Q: What would change if checkpoint credit were sequential, conditional on an earlier checkpoint being correct?

A follow-up from the chat asked whether checkpoint scoring could be made conditional, where a later checkpoint only counts if an earlier one was correct. Yuan explained that OSWorld 2.0 only evaluates the final state of the virtual machine after the agent finishes, not each intermediate step, so checkpoints are graded independently rather than in sequence. She noted that some checkpoints are naturally sequential in practice (you can’t complete a calculation without first collecting the right numbers), but the current scoring doesn’t enforce or check that ordering directly, an agent that gets the beginning wrong but the end right can still score well on the later checkpoints.

Q: How did you decide which YouTube tutorials were useful to build tasks around?

Sid Doorvesh Santbakshsing asked how the team decided which YouTube tutorials were worth turning into tasks. Yuan described a three-part filter. First, the team started from genuinely popular tutorials and discarded any the agent could complete from background knowledge alone, without needing to watch the video. Second, they excluded tasks with alternate, non-GUI solutions, in an app like Blender, an agent that’s good at coding could just script its way to the answer without ever needing to follow the tutorial, which defeats the purpose. Third, they balanced tutorial length against current agent capability: even frontier agents struggle to process video much longer than about 30 minutes, so tasks needed to stay within a range that was hard but not impossible.

Q: If you’ve open-sourced these projects, what parts have you released?

Kazem Jahanbakhsh asked what parts of the project are open source. Yuan said the team released everything: all task code, the full evaluation infrastructure, the code behind all 31 self-hosted websites, and a set of agent rollout trajectories so others can inspect exactly what agents did on these tasks. It’s all on GitHub, and the self-hosted website code in particular means other researchers can build additional tasks on top of the same 31 environments.

Q: Is there a domain you haven’t had the chance to explore yet that you’d like to tackle in a future version?

Vera asked which domains Yuan would like to explore next. Yuan said the team’s focus so far has been on identifying which patterns are difficult for agents in general, which led them toward daily-life and commonly used applications rather than specialized professional tools. Domains like medicine, law, and finance involve expert-specific software that OSWorld 2.0 doesn’t cover, and a future version could expand into those areas, though Yuan noted it’s still an open question whether those domains demand genuinely different agent capabilities or just more of the same ones.

Q: What AI safety considerations or tradeoffs do you have to weigh alongside capability when designing OSWorld 2.0?

Vera raised Anthropic’s note that Claude Fable 5.1 and Claude Fable 5 scored zero on certain tasks due to safeguard interventions, and asked what safety tradeoffs OSWorld 2.0’s design has to account for. Yuan pointed to the same three categories covered in the talk: preventing agents from leaking sensitive information like API keys, penalizing task completions that come with harmful side effects like deleting files, and checking whether agents click through to unsafe lookalike websites without noticing the risk. She agreed this tension, between rewarding capability and penalizing recklessness, is one of the harder open problems in benchmark design generally.

Q: If you had unlimited time and resources, what’s the research question you’d most want to chase?

Closing the session, Vera asked what question Yuan would most want to chase given unlimited time and resources. Yuan’s answer centered on a structural gap in how today’s agents perceive the world. Agents currently operate by taking a screenshot, reasoning about it, and then acting, “like a human closing their eyes to think, then opening them to act.” In a static environment, that’s fine. But OSWorld 2.0 is full of dynamic environments where the world changes between the screenshot and the action, a new pop-up appears, a notification arrives, and the action the agent takes based on its now-outdated screenshot can be technically correct for the state it saw and wrong for the state that actually exists by the time it executes. Yuan sees this as the reason today’s agents struggle with streaming tasks specifically, and as a real limitation of the “look, then act” agent framework more broadly: it isn’t built for a computer that keeps moving while the agent is thinking.

Previous Reading Groups

▶️ OSWorld 2.0: Long-Horizon Computer-Use Benchmark is the eighth session in our ongoing Reading Group series. Catch up on the previous seven:

• Train-to-Test (T²) Scaling Laws
• Senior SWE-Bench: Evaluating Coding Agents
• Agents’ Last Exam
• JudgmentBench: Rubric vs. Preference Evaluation
• Collaborative Gym: Human-Agent Collaboration
• Code World Models & AutoHarness
• Olmix: Data Mixing for LM Development

Speaker bio

Mengqi Yuan is a PhD student at the University of Hong Kong’s XLANG Lab, advised by Professor Tao Yu. Her research focuses on building and evaluating universal digital agents, particularly computer-use agents, and her recent work includes OSWorld 2.0 and OSWorld-Verified. Find her on X and LinkedIn.

Links

• Watch the talk: OSWorld 2.0 | Snorkel AI Reading Group on YouTube (video is currently private, ready to switch to public/unlisted alongside this post)
• Paper: OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks (arXiv:2606.29537)
• Benchmark site: osworld-v2.xlang.ai
• Leaderboard: Snorkel AI: OSWorld 2.0
• Code: github.com/xlang-ai/OSWorld-V2
• Speaker: Mengqi Yuan on X · LinkedIn

Join us for our next Snorkel AI Reading Group, where a researcher presents a recent paper live and takes audience questions. See upcoming sessions and RSVP at snorkel.ai/reading-group.