Terminal-Bench 2.1

Terminal-Bench 2.1 evaluates agent-model systems on 89 technical tasks completed through a command-line interface. The benchmark covers software engineering, machine learning, security, data processing, scientific computing, and system administration. Terminal-Bench 2.1 updates the original benchmark so task failures more reliably reflect agent capability.

Terminal-Bench 2.1 gives the agent a written instruction and access to a purpose-built container containing the relevant files, software, and tools. The agent can inspect the environment, run commands, edit files, install packages, and revise its approach based on execution feedback.

Automated tests then inspect the final state of the container. Terminal-Bench 2.1 accepts any valid approach that produces the required result.

Terminal-Bench 2.1 keeps the 89-task collection from Terminal-Bench 2.0 but repairs 28 tasks where the evaluation could reject valid work. The corrections address changing external dependencies, restrictive resource budgets, and mismatches between written instructions and tests.

Terminal-Bench 2.1 scores should not be treated as interchangeable with Terminal-Bench 2.0 scores. Changes to dependencies, instructions, tests, and resource limits can alter whether the same solution passes. A reliable comparison must run the same agent-model configuration on both versions.

Terminal-Bench 2.1 uses continuous validation to keep its tasks under review after release. New trajectories and community reports can reveal broken dependencies, insufficient resources, ambiguous instructions, or tests that reject valid solutions.

Terminal-Bench maintainers can investigate those cases and update tasks when the evaluation environment causes the failure.

Terminal-Bench 2.1 evaluates the model and agent harness together. The model provides the underlying reasoning and generation capabilities, while the harness controls context management, terminal access, tool execution, and error recovery. The same model can therefore receive different Terminal-Bench 2.1 scores when paired with different agent harnesses.

Terminal-Bench 2.1 marks a trial as successful when automated tests confirm that the final container state satisfies the instruction. The benchmark calculates accuracy from the proportion of successful trials across all tasks.

Terminal-Bench 2.1 runs tasks repeatedly because the same agent-model configuration may succeed on one attempt and fail on another. Confidence intervals represent that run-to-run variation.

Terminal-Bench 2.1 has no unsolved tasks because at least one evaluated agent-model configuration has completed every task. The statement does not mean that every system can solve every Terminal-Bench 2.1 task.

Terminal-Bench 2.1 evaluates a broader range of technical work than SWE-bench. SWE-bench focuses on resolving GitHub issues inside existing software repositories.

Terminal-Bench 2.1 includes workflows such as configuring services, compiling software, training models, processing data, repairing security problems, and reproducing scientific results inside terminal environments.

Researchers can run Terminal-Bench 2.1 with Harbor, the open-source evaluation framework used to execute tasks in containerized environments. Harbor supports established agents and custom agent implementations.

Terminal-Bench 2.1 leaderboard submissions must use the fixed task environments, timeouts, and resource settings. The submission process also requires public trajectories so maintainers can verify the reported results.