OSWorld 2.0

OSWorld 2.0 is a benchmark for computer-use agents working through long, realistic workflows. It contains 108 tasks across seven professional domains, using 31 self-hosted websites alongside desktop applications and authentic input files. Tasks require agents to coordinate information across tools, respond to changing conditions, and leave the computer environment in the correct final state.

OSWorld 2.0 covers research, creative production, engineering, personal services, business and finance, administration and compliance, and healthcare. Example workflows include processing reimbursements, preparing purchase requests, booking travel, completing visa forms, formatting presentations, masking video, and creating CAD models. The tasks represent complete workflows rather than isolated actions inside a single application.

OSWorld 1.0 primarily tests shorter computer tasks, with evaluated agents averaging fewer than 30 steps. OSWorld 2.0 expands the horizon beyond 250 average steps and makes cross-application information dependencies central to most tasks. It introduces authentic artifacts, stateful user information, self-hosted web services, partial-credit scoring, a simulated user, and a separate safety audit.

OSWorld 2.0 reports a binary completion rate and a partial score. Both examine the resulting state of the environment after the agent finishes, rather than requiring one prescribed sequence of clicks or commands.

Partial scoring uses an average of 27.25 checkpoints per task to capture completed portions of the workflow. Most evaluation checks are deterministic, while validated model-based checks contribute 11.53% of the overall score where fixed rules cannot adequately judge the result.

Binary completion records success only when the agent reaches the required final outcome. The partial score gives credit for individual requirements completed along the way. A system can therefore receive a substantial partial score while failing binary completion because one unresolved dependency, missing artifact, or incorrect final action prevents the workflow from being finished.

Live websites can change their interfaces, data, authentication requirements, or availability between evaluation runs. OSWorld 2.0 uses 31 self-hosted, high-fidelity services so each task begins from a controlled state and can be scored reliably after execution. The environments support services such as email, banking, and team communication while preserving access to the open web when a workflow requires it.

OSWorld 2.0 reports results at 150, 300, and 500 agent steps. The 150-step setting tests whether a system can work efficiently, while the 500-step setting gives agents more room to complete long workflows. Results should be compared within the same step budget because additional steps can improve progress while increasing token use and execution cost.

OSWorld 2.0 tests whether an agent can maintain a coherent understanding of a task while the relevant information remains distributed across applications, files, prior records, and user context. Its challenge categories include cross-source reasoning, implicit-state inference, conflict disambiguation, dynamic environments, streaming interaction, multi-item state tracking, and visual-spatial precision.

These tasks can require an agent to notice new information arriving during execution, revise an earlier plan, recover details that were never stated directly in the prompt, or ask the user to resolve conflicting evidence. Successful individual actions are insufficient when the agent loses the context connecting them.

OSWorld 2.0 runs a separate safety audit containing eight diagnostic checks for each trajectory. The safety report is scored independently from task completion, allowing evaluators to distinguish an agent that reaches the requested outcome from one that takes unsafe or inappropriate actions while doing so. Safety-sensitive execution therefore remains visible instead of being absorbed into the main completion score.

Researchers can run OSWorld 2.0 using its open-source repository, which includes the environment framework, model runners, setup documentation, and evaluation utilities. The project currently provides environment images for Docker and AWS. Official task classes are distributed through a gated Hugging Face dataset to reduce benchmark leakage, while the surrounding code and documentation remain public.

Published agent trajectories can be inspected through the OSWorld 2.0 trajectory viewer. Researchers seeking placement on the verified leaderboard must complete the project’s public-evaluation process, which may involve a coordinated evaluation run or submission of monitoring data and trajectories from a trusted institution.