Resource library

The speed of new frontier model releases keeps accelerating. Meanwhile benchmarks struggle to keep up and saturate quickly, often being left in the dust. Most benchmarks are static datasets with no active maintenance, causing them to lose value fast. Some benchmarks are looking to change this by becoming Continuous Benchmarks. Terminal-Bench is one of the most widely reported benchmarks on…

Terminal-Bench 3.0 (formerly Frontier-Bench) recently launched, built to track what AI agents can and can’t do across real computer work. Terminal-Bench 2.1 has been saturating, with top agents reaching 84%; on Terminal-Bench 3.0, the best model, Claude Opus 5, achieves just 43.5%. Terminal-Bench 3.0 raises the bar with 74 authentic, verifiable tasks across 7 domains, designed to expose meaningful gaps…

Parth Asawa (UC Berkeley) presents Continual Learning Bench, the first expert-validated benchmark built to measure whether LLM-based systems genuinely improve with experience, spanning six real-world domains from software engineering to outbreak forecasting.

Nicholas Roberts presents Train-to-Test (T²) scaling laws, which jointly optimize model size, training tokens, and test-time samples—and show why reasoning models should be overtrained well beyond Chinchilla-optimal ratios.

Long-horizon agents operate across many dependent states and transitions, often spanning multiple tools, environments, and periods of external feedback. The difficulty comes from preserving coherent progress as earlier decisions constrain later actions. A single workflow may involve researching evidence, changing files or records, waiting for external responses, revising plans, validating intermediate results, and returning to earlier systems with new information….

Most agent benchmarks still evaluate a thin slice of the job. The agent receives a task, produces an answer, gets scored, and the episode ends. Enterprise workflows work differently. An underwriting agent may need to read policy documents, inspect customer records, call internal tools, ask a simulated user for missing information, update state, and follow approval rules. A correct final…

Anthropic’s Claude Opus 5 recently debuted as the second model overall on the current Senior SWE-bench leaderboard, behind Fable 5. It also achieves the highest score of any evaluated model on the benchmark’s Bug & Performance Investigation category, reinforcing the rapid progress frontier coding models continue to make on increasingly realistic software engineering tasks. Just as notable, Opus 5 reaches…

At our latest Snorkel AI Reading Group, Henry Ehrenberg presented Senior SWE-Bench, an open-source, Harbor-compatible benchmark for evaluating coding agents on realistic, senior-level software engineering work. Its 100 tasks, with 50 public and 50 kept private to mitigate contamination, are sourced from real pull requests across 12 production repositories and cover complex features, migrations, bugs, and performance issues. Senior SWE-Bench…

Text-to-optimization requires two separable capabilities: modeling — choosing the right optimization structure — and binding — grounding every coefficient, index, and parameter in the concrete problem data. We study this via Text2Opt-Bench, a scalable benchmark of solver-verified optimization problems spanning 12 categories, from textbook linear programs to stochastic and multi-objective formulations with up to thousands of variables. Across 10+ models,…