Claude Opus 5: Performance and Error Analysis on Frontier Coding Tasks

Anthropic’s Claude Opus 5 recently debuted as the second model overall on the current Senior SWE-bench leaderboard, behind Fable 5. It also achieves the highest score of any evaluated model on the benchmark’s Bug & Performance Investigation category, reinforcing the rapid progress frontier coding models continue to make on increasingly realistic software engineering tasks. Just as notable, Opus 5 reaches that performance while generating substantially fewer output tokens on average than both Opus 4.8 and several other frontier models evaluated on Senior SWE-bench.

Recent releases across multiple frontier labs – including GPT-5.6 and Claude Opus 5 – continue to raise the ceiling on software engineering benchmarks. But as benchmark scores continue to improve, aggregate rankings tell only part of the story.

In this post, we look at what the latest leaderboard tells us about the current state of frontier software engineering models, why Senior SWE-bench remains an important benchmark, and what a closer analysis of model executions reveals about the different strengths and tradeoffs between GPT-5.6, Opus 4.8, and Opus 5.

Senior SWE-bench starts where many others saturate

As frontier coding models improve, many established coding benchmarks become increasingly saturated. Once leading systems solve the majority of tasks, benchmark scores become less informative for distinguishing meaningful capability differences.

Senior SWE-Bench tasks fall into two major segments: Design-and-Build and Investigate-and-Fix. Design-and-Build tasks focus on the complexity of designing features and migrations without full specifications. Investigate-and-Fix tasks focus on runtime diagnosis of tricky bugs and performance issues from behavioral reports. Senior SWE-Bench feature tasks can be written with natural language instructions that describe the desired feature without a level of specification that would be unrealistic for real-world agent usage, evaluated using a validation agent which uses expert-designed recipes to write system-level, behavioral test scripts adapted to the solutions from agents under evaluation. Bug and performance tasks in Senior SWE-Bench focus on investigation, from starting services to debugging subtle runtime issues. They are sourced from PRs that contain evidence of needing significant runtime investigation to solve (e.g. logs, profiling data, reproduction steps). Rather than just behavioral correctness, Senior SWE-Bench also scores tasteful solves by combining runtime correctness tests with several quality metrics.

Senior SWE-Bench tasks are sourced from real-world PRs merged after February 2026 in open-source repositories, and most tasks are based on PRs authored by engineers with 100+ commits in the respective repository, continuing to differentiate today’s strongest frontier models.

Where Opus 5 leads: 85% on debugging, 67% on builds

A leaderboard tells us how often a model solves a task. It doesn’t tell us how the model approached the problem, where it’s strongest, or why it still fails. To answer those questions, we moved beyond Senior SWE-bench to a separate set of expert-created senior-level coding tasks from our Terminal-Bench+ dataset, built specifically for trajectory-level analysis. Across these tasks, we examined how Opus 5 performs by category, how it compares against GPT-5.6 — evaluated as a pre-release build under early access, before the public Sol, Terra, and Luna variants were designated — and its predecessor Opus 4.8.

The clearest pattern is strength in diagnosis and repair 

Opus 5 solves 85% of debugging tasks and 67% of build-and-dependency management tasks, the categories that most stress a model’s ability to trace faults through an existing system and fix them without breaking it. 

Compared against GPT-5.6, the gap in these areas is wide. Opus 5 solves 85% of debugging tasks against GPT-5.6’s 5%, and leads on build-and-dependency management (67% to 40%) and software engineering (56% to 40%). The two models converge where there’s less headroom, tying on scientific computing (80%) and games (100%), while GPT-5.6 edges ahead only on data-processing (67% vs 60%).

Opus 5’s biggest gains over the previous generation land in these same categories, improving from 25% to 85% on debugging and 33% to 67% on build-and-dependency management versus Opus 4.8. The advances are concentrated exactly where frontier coding work is hardest.

At the task level, Opus 5 wins debugging outright and takes more software engineering tasks head-to-head than both GPT-5.6 and Opus 4.8, while GPT-5.6’s wins are isolated rather than concentrated in any category.

This aligns with what Opus 5 showed on the leaderboard, where it earned the highest score of any model on Senior SWE-bench’s bug & performance investigation category. Two independent evaluations point to the same conclusion: Opus 5’s defining strength is diagnosing and resolving issues in existing software systems.  

Faulty inference causes 35% of failures; tooling just 10%

Pass rates tell us Opus 5 missed roughly four in ten attempts on these frontier tasks, but not why. Using Snorkel’s own failure-mode analysis, we traced every failed trajectory to the specific step where the model went wrong and the error that caused it, producing 77 judge-confirmed root causes across 195 trajectories. 

Faulty inference alone accounts for 35% of failures, which is the single largest mechanism by a wide margin, with output formatting errors and verification errors further down. Operational failures are less common: tool-use, execution, and termination errors together only make up a small share. Opus 5 reliably navigates the environment, calls the right tools, and runs to completion. When it fails it reasons to a wrong conclusion and doesn’t catch the mistake before finishing. 

That distinction matters because tooling and execution failures can often be fixed with better scaffolding. Failures of inference and verification expose a limit of the model itself, exactly what aggregate scores hide. Two models can post similar pass rates while failing for entirely different reasons, and only trajectory-level analysis surfaces the difference. 

“With increasingly difficult and long horizon tasks with multiple objectives, it is imperative to unpack all the ways in which models stumble and their downstream effects on task success. Failure mode analysis of agentic trajectories lets us analyze both branches of information. We can get overviews of why a task failed, pin-point the exact failure steps, and understand how a single point of failure cascades into future steps and affects overall performance; or triggers adaptive retries and new strategies leading to successes in frontier models. This provides us with a lens to also view model strengths through failure analysis.”

The ceiling is reasoning, not scaffolding

Opus 5’s advantage over other frontier models is concentrated in diagnostic and repair work, debugging, and bug investigation, where it outperforms both GPT-5.6 and its own predecessor. Its remaining ceiling is specific and is measurable: when Opus 5 fails, it’s a failure of reasoning and verification, not tooling and execution. A leaderboard tells you how often a model fails. Trajectory-level analysis tells you how and why, which is what you actually need to improve it. 

Explore the frontier coding dataset behind this analysis

Looking ahead

Frontier coding models are improving fast, and Senior SWE-bench measures that progress — explore the full leaderboard. We’re applying the same trajectory-level rigor across the Snorkel Data Series to help teams understand not just how often frontier models fail, but why. Connect with our team to learn more.