Cua-Bench: Benchmarking Computer-Use Agents on Professional Software
Cua-Bench is an open framework for building dynamic, verifiable computer-use environments. Snorkel AI extended it with 25 expert-authored KiCad schematic-editing tasks and found that today’s best computer-use agent can edit an existing circuit, but can’t build one from scratch.
25
Expert-authored KiCad
schematic-editing tasks
built on the Cua-Bench
framework
16%
Full-pass rate for the
best computer-use
agent tested
(4 of 25 tasks)
0 / 16
Build-from-scratch
tasks solved;
every pass was a
single-component edit
What is Cua-Bench?
Cua-Bench is an open-source framework from Cua AI for building verifiable, dynamic computer-use environments, released to fix a specific problem: computer-use agents are wildly inconsistent across minor interface changes. Research has shown task success rates can swing by more than 10x depending on nothing more than an OS theme, font, or language setting because most benchmarks (OSWorld, Windows Agent Arena) rely on static VM snapshots with fixed applications and fixed visual configurations.
Cua-Bench replaces that with a lightweight, declarative framework: tasks are defined in Python via a small set of decorators (scenario config, setup, oracle solution, evaluation), environments run without spinning up full VMs, and the same task can be re-rendered across a dozen OS themes, resolutions, and eras to generate diverse training and evaluation data. It supports desktop (macOS, Linux, Windows) and mobile (iOS, Android) environments, plus adapters for existing benchmarks like OSWorld and Windows Agent Arena.
Why it matters
Most computer-use benchmarks live in the same handful of apps: web browsers, file managers, generic productivity suites. Those evaluations are useful, but they share a structural weakness. The tasks are short, the domain-knowledge bar is low, and the UIs are designed to be intuitive for anyone. A frontier model can often muddle through by trial and error. That isn’t what real professional software looks like.
Snorkel AI used the Cua-Bench framework to build a benchmark around KiCad, a full electronic design automation (EDA) suite with a dense shortcut grammar, multiple linked editors, and objective, machine-checkable ground truth (a schematic either has the right components and wiring, or it doesn’t). It’s a setting that requires real domain expertise, real long-horizon planning, and real GUI fluency at the same time. It also tolerates none of the shortcuts simpler benchmarks accidentally allow.
How the tasks were built
Every one of the 25 tasks was authored by a practicing electrical engineer, and every task ships with three artifacts: a natural-language prompt describing the goal, an initial KiCad project when the task starts from an existing schematic, and a ground-truth final project state that a verifier compares against the agent’s output by reading the netlist.
Tasks were calibrated to take a human expert well under 50 steps, so the agent’s 150-step budget was genuinely generous, not a hidden constraint. A second electrical engineer independently reviewed every task for ambiguity, correctness of the ground truth, and grading fairness. The resulting suite spans single-edit tasks (“change C1 to 2 nF”) through full builds (“design a bipolar-to-unipolar converter from two op-amps, six resistors, and six capacitors”).
Key findings
- A sharp capability cliff. Every task requiring two or fewer component edits was at least partially solvable. Every task requiring three or more components and new wiring either failed outright or earned only partial credit. All four full passes were single-component edits on a pre-existing schematic, not one build-from-scratch task succeeded.
- Failures cluster in planning and perception, not tooling. Across all 25 trajectories, the dominant failure modes were app bring-up and navigation overhead (84% of runs), one-action-per-turn narration that burns turns on self-commentary (84%), zoom/pan oscillation while hunting for off-screen components (76%), and wiring that was simply never drawn (72%). Rolled up to root cause: 40% planning/policy, 22% perception, 19% navigation inefficiency, 11% domain knowledge , as well as zero API errors, meaning the harness worked exactly as expected.
- More steps don’t help. Extending two unfinished runs from 150 to 500 steps didn’t rescue either one; every full pass, by contrast, finished in under 150 turns. The bottleneck is a wrong plan, not a short leash.
- Self-verification is brittle. Five of nine runs that declared themselves complete failed verification, typically because the agent trusted its own narration (“R2 = 16.5k for V_REF = 1.24V”) over ground truth it could have read directly from the netlist, ERC dialog, or title bar.
Snorkel AI’s role
Snorkel researchers Zhengyang (Jason) Qi and Armin Parchami built the 25-task KiCad benchmark on top of the open Cua-Bench framework, with the Cua team providing the underlying harness and verifier support. The analysis, including the capability cliff, the failure-mode taxonomy, the step-budget finding, is Snorkel’s own research contribution to the broader computer-use evaluation effort.
The work feeds directly into CUA-Bench+, Snorkel’s Data Series line for computer-use and GUI agent tasks: containerized desktop environments with ordered action traces grounded in real, professional applications rather than toy productivity software. Snorkel is continuing to extend the EDA suite and run it against additional frontier models.
