Agents' Last Exam v1.1



At a glance
122
55
sub-industries covered
13
40+
academic institutions
Leaderboard
| Rank | Model | Effort | Harness | Pass Rate | Score | Runtime | Output Tokens |
|---|---|---|---|---|---|---|---|
| 1 | GPT-6.2 Astra | Codex |
47.5% |
78.3% | 29h 7m | 2.6M | |
| 2 | Fable 5.2‡ | Max | Claude Code |
46.9% |
72.8% | 227h 20m | 28.2M |
| 3 | Qwen 4† | XHigh | Codex |
46.7% |
74.4% | 185h 6m | 13.6M |
| 4 | Gemini 4 Carbon† | High | ALE-Claw |
45.9% |
77.4% | 156h 47m | 10.2M |
| 5 | Kimi 3.1 Pro | Max | Kimi Code |
43.4% |
74.4% | 61h 26m | 8.5M |
| 6 | Opus 5.5 | Max | Claude Code |
43.4% |
71.6% | 177h 11m | 21.5M |
| 7 | Gemini 4 Hydrogen† | Antigravity CLI |
43.4% |
71.4% | 166h 15m | 25.5M | |
| 8 | GPT-6 Astra | Codex |
42.6% |
73.1% | 80h 12m | 5.1M | |
| 9 | GPT-6.2 Sol | Codex |
41.8% |
72.5% | 31h 40m | 3.1M | |
| 10 | GPT-6.1 Terra | Codex |
41.8% |
69.8% | 124h 36m | 5.5M | |
| 11 | Hunyuan Hy4 | High | Claude Code |
41% |
70.3% | 104h 8m | 19.6M |
| 12 | Gemini 4 Argon† | High | ALE-Claw |
40.2% |
69% | 87h 4m | 6.5M |
| 13 | Muse Wonder 1.0 | High | Muse Code |
38.5% |
68.4% | 116h 27m | 4.7M |
| 14 | DeepSeek V4.1 Flash (1017) | OpenClaw |
36.1% |
65.4% | 76h 51m | 9.3M | |
| 15 | Kimi K3 | Max | Kimi Code |
35.2% |
63% | 107h 48m | 7.5M |
| 16 | GPT-5.6 Sol | Max | Codex |
30.3% |
50.2% | 78h 22m | 3.1M |
| 17 | MiniMax M3.1 Flash | Max | Codex |
29.5% |
58.9% | 117h 37m | 17.4M |
| 18 | MiniMax M3.1 Flash | Max | Claude Code |
29.5% |
57.9% | 122h 12m | 18.1M |
| 19 | GLM 5.3 | Max | ZCode |
26.2% |
54.4% | 111h 10m | 17.4M |
| 20 | GPT-5.6 Luna | Off | Codex |
5.7% |
26.5% | 40h 35m | 1.1M |
‡ Fable 5.2 excludes 9 tasks that are downgraded; its scores are averaged across the remaining tasks.
† Evaluated by external teams.
Key takeaways
GPT-6.2 Astra leads Agents’ Last Exam v1.1 at a 47.5% pass rate and the highest score on the board at 78.3%, using just 2.6M output tokens. Fable 5.2 follows at 46.9% (72.8% score), narrowly ahead of Qwen 4 at 46.7% (74.4% score) and Gemini 4 Carbon at 45.9% (77.4% score). Kimi 3.1 Pro, Claude Opus 5.5 and Gemini 4 Hydrogen follow at 43.4%, then GPT-6 Astra at 42.6%, GPT-6.2 Sol at 41.8%, Gemini 4 Argon (40.2%) and Muse Wonder 1.0 (38.5%). Kimi 3.1 Pro edges out Opus 5.5 on score (74.4% vs 71.6%) with far fewer output tokens (8.5M vs 21.5M), while GPT-6 Astra reaches a 73.1% score with just 5.1M. Fable 5.2’s results exclude 9 downgraded tasks and are averaged over the remaining tasks. Even the top result leaves more than half of the 122 tasks unsolved.
Pass rate vs output tokens
Hover any dot for the harness, model, and exact metrics.
Best result by harness
Top pass rate each harness achieved on this evaluation split, regardless of which model it ran.
Highest Pass Rate observed per harness on this split. Bars are not normalized for sample size; some harnesses were tested across many model variants, others on only one or two.
Industry coverage
Breakdown of the 122 v1.1 tasks by industry. View all tasks on GitHub →
- Business & finance 19
- Computing & math 19
- Health & medicine 19
- Engineering 16
- Physical sciences 14
- Life sciences 12
- Visual media 11
- Psychology & neuroscience 3
- Education & information 2
- Legal 2
- Transport & safety 2
- Other 2
- Social sciences 1

Motion & VFX

3D modeling

Game development

Mold flow analysis

Architectural modeling

Brain imaging
Sample tasks
business_finance
sec_10k_financial_parsing
Parse a SEC 10-K filing into a structured financial schema. Multi-step extraction, table normalization, and cross-reference validation against the original document.
business finance
Health medicine
life sciences
computing math
visual media
Methodology
Metrics
Pass Rate — fraction of tasks the agent fully completed (strict success). Score — average graded outcome across all tasks, including partial credit. Both computed by deterministic graders against hidden references.
Hidden references plus deterministic graders, not LLM-as-a-judge.
Every ~6 months, a new public subset releases with fresh instances. Private tasks rotate into the public pool, retired public tasks rotate out, and held-out private tasks score the official leaderboard, to limit benchmark leakage.
Reference Harnesses
Resources
Behind the benchmark
Agents’ Last Exam (ALE) measures agent performance on long-horizon, economically valuable tasks with verifiable outcomes. The benchmark covers non-physical industries defined with reference to O*NET / SOC 2018 (the U.S. federal occupational taxonomy), spanning 55 sub-industries.
ALE v1.1 ships 122 reference tasks across 55 sub-industries.
Acknowledgments
Agents’ Last Exam is co-led by UC Berkeley RDI and the RDI Foundation, with funding support and contributions from Snorkel AI via the Open Benchmarks Grants program.

