Terminal-Bench 4.0: Why Continuous Benchmarks Require Continuous QA

The speed of new frontier model releases keeps accelerating. Meanwhile benchmarks struggle to keep up and saturate quickly, often being left in the dust. Most benchmarks are static datasets with no active maintenance, causing them to lose value fast. Some benchmarks are looking to change this by becoming Continuous Benchmarks. Terminal-Bench is one of the most widely reported benchmarks on frontier model cards, and I am proud to be a contributor and reviewer.

With the recent release of Terminal-Bench 3.0, now followed by Terminal-Bench 4.0, the team announced an ambitious plan to maintain and develop the benchmark continuously, aiming to keep pace with the frontier. Ryan Marten, the lead on Terminal-Bench, outlined this idea of Continuous Benchmarks in his post. One core idea he shared is that benchmarks are software, and like any piece of software they require active maintenance to preserve their value. Continuous QA is a crucial part of that maintenance. Let’s dive deeper into what continuous benchmarking means in practice going from Terminal-Bench 3.0 to 4.0, and what it takes to build a maintainable frontier dataset.

The core components of QA

The new Terminal-Bench datasets are not just a dump of tasks into a repo, they are the output of a stringent multistage review pipeline. Every stage in it exists because a specific thing tends to go wrong. A task that looks hard can turn out to be trivially gameable. A task that no agent solves can turn out to be unfair rather than difficult. Both look identical from the outside, so the pipeline is built to tell them apart. That pipeline is the backbone of the QA process for developing tasks, and includes the following components:

• Static checks. Structure, formatting, absolute paths, canary strings that keep benchmark data out of training corpora, and a check that the verifier cannot fetch anything at trial time.
• A 35-criteria implementation rubric. Automated review covering everything from whether the instruction alone lets you derive the tests, to whether the task is hard for a real reason rather than tedious.
• Oracle and no-op validation. The reference solution must score 1.0 and doing nothing must score 0.0. Both run in CI on every change.
• Live agent trials. Frontier agents run against the task and reviewers read the trajectories, confirming failures come from genuine capability gaps rather than unclear instructions or a broken environment.
• Adversarial cheat trials. Agents are explicitly prompted to hack the reward before sign-off. Given the chance, an agent will read the test files, fake the output, or edit the verifier, and it will look like a pass. If a task can be gamed, this is where it gets caught.
• Human review and iteration. Maintainers push on difficulty, fairness, and verification quality across multiple hardening rounds.

The pipeline does not only filter tasks, it shapes them. A shortcut found in a cheat trial often means rewriting the verifier to check the final state instead of the method used. A task that no agent solves usually means realigning the instructions with the tests, not making the task easier. Tasks go through several rounds of iteration before final approval, and by the end review has improved the task quality, making them more robust.

Additional QA checks

Once the final dataset was constructed, we still wanted to run more QA rounds and experiment with new ideas to see how tasks behaved as a set. Task-level review can only tell you so much. A failing trial on its own is ambiguous evidence: either the agent could not do the work or the task is broken. I ran a solvability study across the released set to confirm the hardest tasks were still possible to solve. Most were, but a small number were not, and those went back to their authors with specific findings.

Those are the interesting cases, because every one of them cleared the QA gates above. A task can be well-formed and still not be solvable under its own contract. The instructions point one way and the verifier expects something else, or the time budget does not cover the work the reference solution assumes. None of that is a formatting error or a broken environment, and the pre-launch checks missed it. It surfaced later, during additional review.

Every gate above runs before launch, and some issues only surface after. That is normal in any development cycle. The solvability study is just one example of the many systematic benchmark-wide tests I ran. Other things I ran included stress testing reference solutions and verifiers by running tasks in parallel and back to back, which surfaces failures that a single clean run never will.

I reported all of my findings through this process to the task authors and the benchmark team. Just like software development, a backlog forms of issues to fix and improvements to make. Some fixes land sooner and some later, which is exactly why the team built the new versioning system to release benchmark updates periodically. Many of the issues I found were fixed before 3.0 shipped, some are only live now with 4.0, and more will come in later versions.

This rigorous process with a community providing feedback across the full benchmark and a dedicated process for managing and fixing issues is how this benchmark will remain relevant and earn the trust of the industry.

What changed from 3.0 to 4.0

Versioning is not only about shipping fixes. A new version is the opportunity for a full leaderboard re-run with the latest changes, which makes it the moment to change anything that touches every task. 4.0 batches a lot of work together, including:

• Base and built image pins for reproducibility
• Timeout increases
• CPU and memory increases
• Task fixes
• Task removals

Some of these changes were benchmark-wide, like the resource increases. During testing it was found that some tasks were not failing because agents could not do the work, they were failing because the budget did not cover it. That is a benchmark-wide calibration problem rather than a task bug, and you only see it by looking at what agents did across the whole set once the initial 3.0 dataset was complete. Now the team is taking these learnings and applying them to make 4.0 a better version of the benchmark.

What comes next

The next step is taking what additional QA surfaces and folding it into the QA process for next time. There is a continuous cycle here. As issues are discovered, we can find ways to build automated checks to catch them going forward. We can also put agents to work on it, proactively hunting for new issues and developing evals to catch them. Improving the data by continuously improving the QA is how we accelerate the benchmark development cycle and keep pace with the frontier.

Just like benchmarks, Snorkel builds our frontier datasets, including Terminal-Bench+, the same way. Expert-built, programmatically verified, continuously maintained. Connect with our team.