Terminal-Bench 4.1 hardens the verifiers behind the scores

Terminal-Bench 4.1 is now available. Snorkel AI contributes to Terminal-Bench as a task author, a data partner, and a supporter through the Open Benchmarks Grants program. The release keeps the same 66 tasks, with verifier fixes, reproducibility improvements, hardening against reward hacking, and a new 63-task CPU-only dataset.

Terminal-Bench and real terminal work

Terminal-Bench measures whether an AI agent can use a computer terminal to complete software engineering and systems tasks, including writing and debugging code, configuring environments, and recovering from failure. Terminal-Bench 4.1’s 66 tasks span seven domains: software, science, machine learning, operations, security, hardware, and media. The benchmark is hosted by Harbor and the Laude Institute.

Each task has three parts: an instruction, an environment, and a verifier. The instruction describes what the agent must accomplish. The environment supplies the files, tools, and resource limits available to the agent. The verifier inspects the result and returns a reward, which contributes to the reported score.

Terminal-Bench is used to report frontier agent performance and is commonly cited across model cards. The Terminal-Bench 4.1 leaderboard lists 23 model and agent configurations, with resolution rates from 12.1% to 65.8%. Seven of them resolve more than half of the tasks. Opus 5.5 is highest at 65.8%, followed by Sonnet 5.5 at 63.9% and GPT-6.1 Sol at 60.9%. The leaderboard also reports cost and token use showing some variability amongst models, with Opus 5.5 using 8.0B tokens at $4.7k and GPT-6.1 Sol using 1.5B tokens at $0.6k.

Five challenges that affect agentic benchmarks

Many agentic benchmarks face recurring challenges, especially as stronger models run against tasks at scale:

1. Saturation. When top models solve a task on every attempt, it stops separating models. As models improve, benchmarks need to identify tasks that no longer provide a useful signal.
2. Environment noise. Runs have limits on time, CPU, and memory. If those limits are too tight, failures may reflect the sandbox rather than the agent.
3. Verifiers that can be gamed. If agent-supplied code can reach the grader or its reward output, it may be able to tamper with the evaluation instead of completing the task.
4. Verifiers that are too strict. A grader can reject valid work, for example when a task’s instructions and tests disagree or a check is flaky. Terminal-Bench addresses these cases as users and leaderboard runs surface them.
5. Changes that make results hard to compare. Updates to tasks, environments, or verifiers can affect whether scores from different releases are comparable. Clear versioning and change histories help readers interpret results.

These challenges can produce misleading results in different ways: a hacked verifier can return a plausible pass, an overly strict verifier can return a plausible failure, and untracked changes can make a leaderboard appear consistent while measuring different things.

Terminal-Bench 4.0 addressed saturation and environment noise

Terminal-Bench 4.0 reduced the task set from 74 to 66. It removed eight tasks: two that were saturated, two that triggered model refusals, two with public solutions, and two with unresolved quality or platform-compatibility issues. For that release, a task counted as saturated when every model class in every family of the latest model generation solved it on all five attempts. No tasks were added.

The 4.0 release fixed 19 tasks. The updates changed instructions, environments, or verifiers to address flakiness and misspecification reported by users or observed in leaderboard runs. Each task’s README includes its own changelog.

The 4.0 release also recalibrated resource limits following Anthropic’s methodology for measuring infrastructure noise in agentic coding evaluations. Tasks were run with substantial headroom, then settings were selected that did not change the final metrics. Every task received an eight-hour agent timeout. This reduced agent timeouts and errors compared with 3.0; remaining errors were largely model refusals and output-token-exceeded errors.

With fewer failures caused by resource limits, attention turns to how reliably the verifier judges the work an agent completes.

Terminal-Bench 4.1 strengthens verifier isolation

Terminal-Bench verifiers can execute agent-supplied artifacts inside the verifier container. If submitted code runs with access to the grader or its output, it may be able to interfere with the grading process. Fixing individual known exploit patterns can help, but each added check can also reject valid work.

Terminal-Bench 4.1 adopts RewardKit Workers, part of Harbor’s recent work on reward hacking, to strengthen isolation in affected verifiers. RewardKit is an optional package from the Harbor team that simplifies verifier creation. A Worker runs submitted code as an unprivileged user, prevents it from reading the tests or writing the reward, and stops processes it started before returning a result. This strengthens the grading boundary; it does not replace broader sandbox and resource controls.

Even with that separation, task authors need domain expertise to decide whether the checks capture successful completion and allow legitimate alternative solutions. Improving grading correctness was also part of the release, so 4.1 includes verifier fixes alongside the isolation changes.

The release also adds reproducibility improvements: prebuilt release images pinned by digest for task environments, verifiers, and Compose services, plus pinned container images and locked task dependencies. It also adds a CPU-only dataset of 63 tasks, published with the same release tag and task revisions as the full dataset. No tasks were added or removed, but all 66 tasks were modified, so results should identify the exact benchmark release. Re-grading the 4.0 leaderboard trials with the 4.1 verifiers flipped about 1% of rewards, and leaderboard scores remained within previously reported confidence intervals.

Structuring a robust benchmark

Terminal-Bench publishes its tasks, verifiers, solutions, and resource settings, and triages issues in the open. Public task histories let researchers inspect what a verifier checks, review proposed fixes, and see how the benchmark changes over time.

A reported weakness is most useful when it is connected to the runs it could affect. Reproducing an exploit establishes the conditions under which it works, while examining recorded trials shows whether agents actually encounter those conditions and how the exploit changes their scores. The 4.1 re-grade applies the same idea to fixes, measuring what the updated verifiers changed in the recorded 4.0 trials instead of assuming the effect. That kind of evidence helps reviewers assess the impact on current results while still addressing vulnerabilities that have not yet appeared in ordinary runs.

A snapshot of open issues describes where a benchmark stood on a given day. How it investigates, tests, and fixes reported problems is important to establish trust in the leaderboard. Snorkel AI takes part in this process, and the maintainers acknowledged its work auditing Terminal-Bench at launch. The audit will be shared separately in the near future.

The takeaway

A benchmark score is a claim about capability, and the verifier is what makes that claim checkable. Terminal-Bench 4.0 addressed task saturation and environment noise. Version 4.1 focuses on verifier fixes, resisting reward hacking through better isolation in affected grading paths, and improving reproducibility. When an updated verifier changes a score, researchers should examine which judgments changed in the underlying runs. Understanding whether the fix rejects a shortcut or accepts previously rejected valid work helps explain what the revised score measures.

Interested in learning more? Explore the Terminal-Bench 4.1 release and dataset, or read the RewardKit documentation.