Grok 4.6: Cost vs. Performance On Senior SWE Tasks

---

Main Takeaway

Note: All figures reflect the leaderboard as of August 20, 2026, scored over the 95 tasks in the current evaluation set (100 benchmark tasks, 5 excluded for reward hacking).

The Benchmark

Senior SWE-Bench, developed by Snorkel in collaboration with researchers at Princeton and UW–Madison, is a benchmark designed to evaluate agents on senior-level software engineering work. It scores every attempt in two ways.

A basic solve requires that a submission pass all applicable runtime correctness checks, including prewritten verifiers, the validation agent’s synthesized behavioral tests, or both, depending on the task. A tasteful solve requires what the basic solve does plus four quality gates: a rubric score above 0.5, patch bloat under 2× the reference implementation, a relative-taste score above 2/5, and adherence to the repository’s existing practice above 2/5. The taste and practice scores come from an LLM judge panel calibrated against SWE expert reviews. Taken together they serve as a shorthand for the question a senior reviewer would ask, “would this merge?”, rather than a direct measurement of merge decisions.

The best tasteful pass@1 score on the current board is 34.7%, held jointly by Fable 5, Opus 5, and GPT-5.6 Sol, providing valuable headroom where other measures saturate.

Methodology

All models run on the Mini-SWE-Agent harness via Harbor, the open-source framework for LLM/agent evaluations, with three attempts per task, reported three ways: pass@1 (the average single attempt), pass@3 (at least one of three succeeds), and pass^3 (all three succeed). Figures for Grok 4.6 are at high effort.

Grok 4.6 used an average of 23.9K output tokens per task. At the published price of $6 per million output tokens, that corresponds to approximately $0.14 in output-token cost per task. This estimate excludes input-token costs, cached context, validation overhead, and harness-level retries, so it should not be interpreted as total evaluation or deployment cost.

Three trials per task is enough to report pass@3 and pass^3, but not enough to establish that the small differences between models are statistically meaningful. The rankings and point gaps mentioned are leaderboard observations under this configuration; they should not necessarily be read as establishing that one model is superior to another.

Solve Rates and Cost

Grok 4.6 solves 65.3% of tasks at the basic bar (pass@3), placing second behind Opus 5 (70.5%) and marginally ahead of Fable 5 (64.2%). At the tasteful bar it reaches 38.9%. Every model on the board gives up ground between those two bars because the tasteful rubrics are especially demanding with none of the frontier models clearing more than 46.3% of tasks.

What makes this placement notable is its cost. Grok 4.6 hits 8.4 points behind the three-way tie at the top of tasteful pass@1, 34.7% held jointly by Fable 5, Opus 5, and GPT-5.6 Sol, while its output-token cost is 7.0× cheaper than Sol, 12.6× cheaper than Opus 5, and 20.9× cheaper than Fable 5. Its 23.9K output tokens per task is lower than nearly every other frontier configuration evaluated.

Plotted against output-token cost, Grok 4.6 sits on the Pareto frontier at both low and high effort. Much of the field cannot say the same with a cluster of configurations between $1 and $6 per task dominated on both axes.

Scatter plot of tasteful pass@1 solve rate against average output token cost per task, log scale, for all 16 models at every effort tier. Solid dots mark the Pareto frontier.

Consistency Across Attempts

Grok 4.6 ranks fourth on tasteful pass^3, where all three attempts on a task must succeed. Its 16.8% score is 1.1 percentage points below Opus 5 at high effort and above several models that score better on average single-attempt performance, including GPT-5.6 Sol (12.6%), Claude Sonnet 5 (14.0%), and GPT-5.5 (10.5%), each at 7–19× the output-token cost. These results suggest that Grok 4.6’s relative position improves under the benchmark’s stricter repeated-success criterion, although three attempts per task may not be sufficient enough to draw strong conclusions about reliability beyond this evaluation.

Scatter plot of tasteful pass^3 solve rate against average output token cost per task, log scale, showing the Pareto frontier.

SpaceXAI describes Grok 4.6 as building on Grok 4.5 through a longer supplemental training run, regenerated supervised fine-tuning trajectories, and reinforcement learning across agentic tasks. Those changes provide relevant context for the model’s results, but this evaluation does not isolate which training interventions produced the observed gains.

Conclusions

Senior-level engineering work remains an open frontier with even the strongest models on the board clearing the tasteful bar on fewer than half of all tasks. Against that baseline, Grok 4.6 places second of sixteen on basic solve rate, fourth on tasteful pass^3, while incurring a fraction of the output-token cost of the models ranked above it.

On Senior SWE-Bench, Grok 4.6 offers a strong cost-performance tradeoff: it comes within single digits of the leading tasteful pass@1 score while incurring roughly an order of magnitude less output-token cost than several top configurations. Whether that advantage generalizes to other harnesses, workloads, pricing conditions, and total-cost calculations requires further evaluation.

We will continue publishing leaderboard updates and analysis as new frontier models are released. The full leaderboard and per-agent breakdowns are available on Senior SWE-Bench (https://senior-swe-bench.snorkel.ai).

---

Explore more Snorkel leaderboards: Terminal-Bench 4.0, Terminal-Bench-Science, Terminal-Bench 3.0, OSWorld 2.0, Agents’ Last Exam, Continual Learning Bench, and SlopCode Bench. See the full list at snorkel.ai/leaderboard.