Terminal-Bench 4.0
A benchmark to measure and evolve with the frontier of agent work: real terminal environments, real software engineering tasks, and a rolling task set that is revised as agents catch up to it. Version 4.0 trims the 3.0 task set from 74 to 66 tasks, removing eight and revising 20.
At a glance
66
8
20
Terminal-Bench 4.0 Pareto Frontier
- Opus 5.5
- Sonnet 5.5
- GPT-6 Astra
- GPT-6.1 Sol
- Fable 5.1
- Opus 5
- GPT-6 Sol
- Fable 5
- GLM-5.3
- Grok 4.7
- GPT-5.6 Sol
- GLM-5.3-Flash
- Qwen3.8-Max-0902
- Opus 4.8
- GPT-5.6 Terra
- Grok 4.6
- Gemini 3.8 Flash
- GPT-5.6 Luna
- GPT-6 Luna
- Muse Spark 1.3
- Grok 4.5
- Sonnet 5
- Gemini 3.7 Flash
Leaderboard
| Rank | Model | Effort | Agent | Resolution Rate | Release Date | Tokens | Cost |
|---|---|---|---|---|---|---|---|
| 1 | Opus 5.5 | max | Claude Code |
64.8%
±3.1
|
2026-09-22 | 8.0B | $4.7k |
| 2 | Sonnet 5.5 | max | Claude Code |
61.8%
±2.9
|
2026-09-28 | 19.4B | $7.3k |
| 3 | GPT-6 Astra | max | Codex |
58.2%
±2.8
|
2026-09-03 | 1.5B | $3.3k |
| 4 | GPT-6.1 Sol | max | Codex |
58.2%
±3.1
|
2026-09-29 | 1.5B | $0.6k |
| 5 | Fable 5.1 | max | Claude Code |
57.9%
±3.8
|
2026-09-01 | 2.7B | $6.2k |
| 6 | Opus 5 | xhigh | Claude Code |
53.9%
±3.2
|
2026-07-24 | 6.9B | $6.1k |
| 7 | GPT-6 Sol | max | Codex |
49.4%
±3.2
|
2026-09-22 | 5.0B | $1.5k |
| 8 | Fable 5 | max | Claude Code |
44.5%
±3.8
|
2026-06-09 | 3.8B | $7.3k |
| 9 | GLM-5.3 | max | Claude Code |
41.8%
±3.2
|
2026-08-14 | 8.7B | $2.7k |
| 10 | Grok 4.7 | xhigh | Grok Build |
37.6%
±3.5
|
2026-09-21 | 5.5B | $3.7k |
| 11 | GPT-5.6 Sol | max | Codex |
37.3%
±3.8
|
2026-06-26 | 4.4B | $2.5k |
| 12 | GLM-5.3-Flash | none | Claude Code |
35.8%
±3.5
|
2026-08-26 | 2.9B | $2.9k |
| 13 | Qwen3.8-Max-0902 | max | Claude Code |
27%
±3.8
|
2026-09-02 | 3.0B | $4.6k |
| 14 | Opus 4.8 | max | Claude Code |
23.6%
±3.6
|
2026-05-28 | 6.4B | $6.5k |
| 15 | GPT-5.6 Terra | max | Codex |
21.5%
±3.3
|
2026-06-26 | 5.7B | $1.7k |
| 16 | Grok 4.6 | high | Grok Build |
20.3%
±3.1
|
2026-08-12 | 4.0B | $3.6k |
| 17 | Gemini 3.8 Flash | high | mini-SWE-agent |
19.1%
±3.4
|
2026-09-02 | 17.2B | $1.8k |
| 18 | GPT-5.6 Luna | max | Codex |
17.3%
±2.8
|
2026-06-26 | 11.6B | $0.3k |
| 19 | GPT-6 Luna | max | Codex |
16.4%
±2.7
|
2026-09-22 | 3.9B | $0.1k |
| 20 | Muse Spark 1.3 | xhigh | Muse Code |
14.5%
±2.9
|
2026-09-02 | 9.0B | |
| 21 | Grok 4.5 | high | Grok Build |
12.4%
±2.6
|
2026-07-16 | 3.4B | $2.1k |
| 22 | Sonnet 5 | max | Claude Code |
12.4%
±3.1
|
2026-06-30 | 21.6B | $9.6k |
| 23 | Gemini 3.7 Flash | high | mini-SWE-agent |
11.2%
±2.4
|
2026-08-13 | 11.1B | $1.3k |
How Terminal-Bench 4.0 compares
Benchmark
Released
Tasks
Change
Terminal-Bench 4.0
Aug 2026
66
8 tasks removed, 20 revised, 0 added
Terminal-Bench 3.0
Jul 2026
74
New rolling task set (formerly branded Frontier-Bench); initial 74 tasks added
Terminal-Bench 2.1
May 2026
89
Revision fixing 28 tasks; task count unchanged
Terminal-Bench 2.0
Nov 2025
89
Harder, curated task set; introduced Harbor, the agent eval and optimization framework
Terminal-Bench 1.0
May 2025
80
Initial release (Terminal-Bench-Core), led by Stanford and the Laude Institute
Methodology
Resolution rate
The public Terminal-Bench site marks its task and leaderboard data with a canary string, requesting that benchmark data not appear in model training corpora.
Run command
Evaluated via the Harbor evaluation harness: harbor run -d terminal-bench/terminal-bench@4.0.0. Dataset pinned at hub.harborframework.com.
From the blog
Resources
Behind the benchmark
Terminal-Bench measures whether an AI agent can operate a real terminal to complete real software engineering and systems tasks: writing and debugging code, configuring environments, and recovering from failure, all without a GUI. Version 3.0 introduced the current 74 task families; version 4.0 is a maintenance release that removes eight tasks and revises twenty, netting 66 tasks, with no new tasks added in this cycle.
Acknowledgments
Terminal-Bench is hosted by Harbor and the Laude Institute, and supported by Snorkel AI via the Open Benchmarks Grants program. The v4.0.0 release was authored by Ryan Marten and Snorkel's Justin Bauer is among the benchmark's senior reviewers.
