Terminal-Bench 3.0

Terminal-Bench 3.0 is an open benchmark for evaluating whether AI agents can complete valuable work in containerized terminal environments. Developed initially as Frontier-Bench, it covers tasks across technical and professional domains and is designed to remain useful as terminal agents improve.

Before a task reaches the leaderboard, it must pass implementation checks, solvability review, agent trials, adversarial testing, and human review. A successful reference run is one input to that process.

Each contribution moves through several gates. Static checks inspect the canary, Dockerfile, paths, metadata, and other implementation requirements. An automated rubric reviews 35 criteria, including whether the task is solvable, verifiable, appropriately difficult, and resistant to cheating. Oracle and no-op runs then confirm that the reference solution succeeds and an empty attempt fails.

Tasks that pass those checks receive maintainer review, agent trials, and cheat trials. If an adversarial agent finds a shortcut through the grader, the task returns to a hacker-fixer loop before merge. The later stages catch cases where a task is technically valid yet the grader rewards a shortcut or incomplete solution.

Each task belongs to one of seven fixed top-level domains and a more specific subdomain. The primary skill required to complete the task determines the category. Libraries and tools used along the way remain secondary. The top-level domains stay stable while subdomains expand as contributors add new kinds of work.

The benchmark keeps task execution and grading in separate environments, limiting the files and state available to the verifier. Cheat trials then ask an adversarial agent to search for grader exploits.

When a trial exposes a shortcut, the author or maintainer hardens the environment or verifier and runs the attack again. Repeated adversarial review tests known failure paths and gives maintainers a process for examining new attacks as agents change.

Yes. Start with the contribution guide and task-proposal rubric, then ask for feedback on the idea before building the full environment. Tasks are submitted through pull requests, where automated checks, maintainer review, agent trials, and adversarial testing guide revisions toward acceptance.

Snorkel AI contributes tasks and supports Terminal-Bench 3.0 as a data partner through its Open Benchmarks Grants program. Snorkel researcher Justin Bauer is also listed among the benchmark's reviewers.

Terminal-Bench is hosted by Harbor and the Laude Institute. Snorkel contributes task development, data partnership, grant support, and review. Benchmark ownership remains with its hosts; Snorkel's role is contributory.

Terminal-Bench 3.0 is the next generation of the Terminal-Bench task family. It expands the domain mix, accepts new community-authored tasks, and adds adversarial review intended to keep tasks useful as agents improve. Terminal-Bench 2.1 remains a separate, continuously validated release with its own task set and leaderboard.