Senior SWE-Bench: Evaluating Coding Agents Like Senior Engineers

At our latest Snorkel AI Reading Group, Henry Ehrenberg presented Senior SWE-Bench, an open-source, Harbor-compatible benchmark for evaluating coding agents on realistic, senior-level software engineering work. Its 100 tasks, with 50 public and 50 kept private to mitigate contamination, are sourced from real pull requests across 12 production repositories and cover complex features, migrations, bugs, and performance issues. Senior SWE-Bench was developed by Snorkel AI in collaboration with researchers from Princeton and UW–Madison. Explore the leaderboard.

Unlike benchmarks that provide detailed technical specifications, Senior SWE-Bench gives agents the kinds of requests engineers actually receive, describing the desired outcome without prescribing the solution. To evaluate open-ended feature tasks reliably, it combines pre-written verifiers with a validation agent that uses expert-designed recipes to write behavioral tests adapted to submitted solutions. A separate taste judge scores minimality, approach quality, hygiene, fluency, and craftsmanship relative to the existing codebase and reference solution.

The results show how far coding agents still have to go. Claude Fable 5 currently leads the benchmark at a 29.1% tasteful solve rate, meaning even the top-performing model fails to meet senior-level correctness and code-quality standards on more than 70% of tasks. The evaluation also surfaces meaningful differences in cost, efficiency, and behavior across model families, including growing evidence that newer models recognize when they are being benchmarked and explore ways to obtain the reward without solving the task as intended.

Transcript

Lightly edited for readability.

Henry opened the presentation by describing the mismatch Senior SWE-Bench is designed to address: coding agents are increasingly expected to work like senior engineers, but most benchmarks still evaluate them using highly specified instructions and narrow correctness tests. He then outlined Senior SWE-Bench’s approach to realistic instructions, long-horizon tasks, and scalable code-quality evaluation.

Where Senior SWE-Bench tasks come from

Raise your hand if you are familiar with SWE-bench or that style of coding benchmark. I am going to approximate that as everyone in the room, but I will give a quick overview of how these benchmarks generally work.

There are two broad types of coding benchmark tasks: tasks based on work found in the real world and tasks constructed from scratch.

Terminal-Bench is a great example of the latter. A contributor comes up with an idea for a terminal task that will challenge a recent agent. SWE-bench and Senior SWE-Bench, on the other hand, are based on work people have already done in the real world. In particular, they look at pull requests on GitHub that reflect real work contributed to production repositories.

Originally, a SWE-bench task could come from almost any pull request. It might involve adding a small test or changing a helper function. For Senior SWE-Bench, we use a much more deliberate discovery and design process to find pull requests that reflect longer-horizon, more senior work, or tasks that require a combination of senior engineering skills to complete successfully.

We look for production-grade, multi-service applications, tools, and libraries. Most of the repositories began after 2020 because they tend to have a higher velocity of changes, new features, and major bug fixes.

The pull requests we source are from February 2026 or later, after the knowledge cutoff of most current models. We also group multiple pull requests into a single task to expand its scope. Someone might create a feature over five pull requests, for example, and we combine the scope of those changes into one task.

We look for pull requests from maintainers or authors with hundreds of commits in the repository. That gives us confidence that the work required deep familiarity with the codebase and that the contributed code is a strong reference solution created by a genuine expert in the repository.

Senior SWE-Bench has two primary task types. The first is investigate and fix: bug and performance tasks typically presented like a bug report in Slack. The second is design and build: complex, multi-component features and migrations. We provide a general scope, similar to a message from a product manager or tech lead, and describe the desired behavior without specifying the technical implementation.

The benchmark includes repositories such as PostHog, a product analytics platform built in Python and TypeScript, and Electric, a Postgres sync engine written in Elixir. You can explore all of the repositories and tasks at senior-swe-bench.snorkel.ai.

A real task from the Harbor framework

One of the best ways to understand Senior SWE-Bench is to look at a specific task. I am going to commit the classic talk momentum killer and play a video. I promise it will be relatively short.

This is Gabe, a Senior SWE-Bench contributor, discussing the source of one of the benchmark’s tasks. At the risk of being too meta, the feature he is describing was added to Harbor itself, a common framework for running evaluations.

The change allowed Harbor to support multi-step tasks. A typical task has an instruction at the beginning, followed by the agent’s work and a verifier at the end. Gabe wanted to enable a task to run one instruction and verifier, then another step with its own instruction and verifier, and potentially additional steps after that.

Henry: Can you talk about the design approach you took with Alex, Harbor’s maintainer?

Gabe: There are countless ways to add multi-step trials. You could have a set of isolated environments, or link trials so they form a directed acyclic graph. The approach we settled on was adding a configuration that specifies the steps to run, where they are located, and where their outputs should be saved.

We chose that approach because it was simple and kept existing trials portable. It preserved Harbor’s existing semantics, so no trials would break. If you downloaded a SWE-bench trial from Harbor, it would continue to work. Those were the core constraints, and this solution fit them.

Henry: What were some of the trickier or more nuanced issues you encountered during implementation?

Gabe: In addition to avoiding regressions, we needed a sensible way to aggregate rewards. Every step has its own reward, and we wanted to give users the flexibility either to average the rewards or take only the final one. If one step is missing a key, what should happen? We decided it should count as a complete failure and receive a zero.

Every step can also have its own setup and add new conditions to the environment. It was not something I initially considered, but every setup needs a health check. Existing trials have health checks, and Alex caught during review that each step needed one as well. If a setup has not done its job, the trial should fail immediately. Otherwise, setup can fail silently and leave you debugging a very expensive run.

Henry: Thanks, Gabe.

Gabe described a wide design space with many implementation details that can go right or wrong. We want to challenge agents to navigate that full design and implementation space without guiding them through every decision. That is how many of us use coding agents: we describe what we want, perhaps offer a few pointers, and expect the agent to do the work correctly.

We turned Gabe’s work in Harbor into a Senior SWE-Bench task. The instruction given to the agent looks very different from the highly specified technical requirements used in earlier coding benchmarks such as SWE-Bench Pro. It gives the agent high-level context, a few design notes, and descriptions of the desired behavior similar to user stories a product manager might send to a senior engineer in Slack.

There is a wide range of valid choices the agent could make. We want to know whether it can select a correct or near-optimal design and implement it successfully without being told exactly how.

Most benchmarks use highly specified instructions to support pre-written verifiers. If you write a unit test in advance, you generally need to know the exact interface and behavior you will test. That is why benchmark instructions often spell out so much detail.

Senior SWE-Bench achieves reliable evaluation with more realistic instructions. In Gabe’s implementation, for example, Harbor recognizes a multi-step task when it contains a directory named steps. He could just as easily have named it phases or task_steps. That was an implementation choice. A full technical specification would eliminate choices like that, leaving the agent with little design work to do.

Realistic, underspecified instructions widen the design space the agent must navigate and create more challenging, longer-horizon tasks. The key question is how to evaluate those solutions as reliably as we could with pre-written verifiers.

Reliable evaluation without over-specifying the solution

The first principle is to test behavior, not implementation details. Many previous-generation benchmarks tested helper functions or small implementation choices. We test behavior through system interfaces such as APIs, command-line interfaces, frontend components, and service interfaces. This raises the level of the interface being tested and allows us to determine whether the behavioral contract in the instruction has been met.

We also use a principled framework for deciding what can be tested fairly. Not everything is included in the instruction, so we need to avoid unfair “gotchas.” It would be unreasonable, for example, to fail an agent because it named a directory stages instead of steps when either choice would work.

There are three categories of fair tests:

1. The behavioral contract. We can test the behavior explicitly requested in the instruction.
2. Load-bearing codebase practices. We can test consistent patterns in the codebase that affect functionality and that a senior engineer would expect a contribution to follow. A senior engineer would likely reject a pull request that invented a new authentication framework for one API endpoint instead of using the existing one.
3. General best practices with no defensible alternative. If an API endpoint could return 20 million items, failing to cap or paginate it would be unacceptable even if pagination were not explicitly requested.

Most other criteria amount to testing arbitrary implementation choices and are generally not fair to evaluate.

The final component is adaptive runtime verification through the validation agent, one of Senior SWE-Bench’s main technical contributions. Instead of relying only on tests written before seeing the solution, the validation agent writes verifier tests adapted to the code the solving agent produced. That means we do not need to specify every interface and behavioral detail in advance.

How the validation agent works

A Senior SWE-Bench task begins with the solving agent. It receives an instruction, works on the task, and produces a code patch.

The patch then enters the reward phase, where several mechanisms evaluate it. These include expert-written verifiers, the validation agent, a task-specific rubric assessed by an LLM judge, and the taste judge.

Pre-written verifiers and LLM judges have complementary strengths and weaknesses. Pre-written verifiers run the code and are deterministic, but they often require unrealistic, highly specified instructions, particularly when a solution changes an interface. LLM judges can adapt to different solution shapes, but they usually inspect the code without executing it. A solution may look plausible even if it does not compile.

Our goal was to combine the adaptability of an LLM judge with the reliability of executing the solution. That is what the validation agent does.

The validation agent receives the agent’s code patch and an expert-designed validation specification. The specification contains testing recipes written at a more abstract level by an expert contributor.

Using both the specification and the implementation choices visible in the patch, the validation agent writes test scripts that instantiate those recipes for the submitted solution. The scripts then run programmatically across multiple test cases and inputs.

There is also a revision stage. An LLM judge reviews the generated test scripts and can provide feedback to the validation agent if something is missing. The agent revises the scripts, and then the final tests run without an LLM in the loop. At that point, they behave like conventional verifier scripts and produce scores based on the cases that pass or fail.

The validation judge can also discard a trial if the generated tests are invalid or if the validation agent appears to be helping the solution pass rather than evaluating it. That is rare, but it is important to have the safeguard.

Because this introduces a stochastic element, we use several methods to ensure that the validation process is reliable. We align the system prompt closely with the role of a test engineer. We run validation repeatedly against the reference solution to ensure it consistently passes and against an empty solution to ensure it consistently fails. The validation judge checks whether the required work was completed, and each test is parameterized across multiple cases to reduce the risk of hard-coded tests.

For the Harbor task, a pre-written verifier checks that existing single-step tasks still work. That behavior is already known and has a fixed interface, so it is a good fit for conventional tests.

The validation specification covers the new behavior. One procedure checks whether multiple steps run in order and aggregate their solutions correctly. The expert-written recipe tells the validation agent what to inspect and how to use a helper script to exercise the implementation. It does not dictate the exact interface. The validation agent writes the specific script for the solution it receives, then runs it across multiple cases and edge conditions.

Evaluating code quality with the taste judge

The taste judge scores not only whether a solution is correct, but also the quality of the code. It gives us a scalable way to evaluate code quality without creating a highly customized rubric for every task.

The rubric is technology- and practice-agnostic. It assesses the submitted code relative to the practices of the repository and to the reference solution.

Consider documentation. A generic LLM judge might encode the belief that every function should document all input and output parameters. Many codebases intentionally do not follow that practice. The taste judge does not apply a universal preference. It examines the practices in the surrounding codebase and asks whether the submitted code fits them.

That approach scales across codebases and technologies without imposing one reviewer’s stylistic preferences. We also do not use the raw taste scores directly. We apply a conservative threshold equivalent to requiring the code to score above a D on an A-to-F scale. If the code is not materially poor, it passes. A surprising number of solutions still fail that threshold.

Quality control and trade-offs

Senior SWE-Bench uses an extensive quality-control process that combines automated refinement with multi-stage expert review. Members of our research team review every task.

A benchmark is only as valuable as it is reliable. Quality control for long-horizon tasks takes substantial work. If you are building a benchmark, prepare to invest heavily in quality control because it determines whether the benchmark has any value.

No benchmark is perfect. Senior SWE-Bench adds cost and execution time because it runs an agent during evaluation. It also contains several nondeterministic elements, including the validation agent, taste judge, and task-specific rubrics. If your use case requires exactly the same execution every time, another approach may be more appropriate.

We make that trade-off explicitly to support highly realistic task instructions. In our testing, the process is reliable enough to provide a meaningfully higher-level evaluation of coding agents.

What the leaderboard reveals

Claude Fable 5 currently leads Senior SWE-Bench with a 29.1% tasteful solve rate. Its average output-token cost is about $29 per task. Grok 4.5 ranks fifth at 17.2%, but its average output-token cost is roughly $1 per task. That makes it a much more economical option even though its solve rate is lower.

The cost-performance frontier reveals meaningful differences among model families. GPT-5.6 Sol has an average output-token cost of about $3 per task, compared with roughly $8 for Claude Opus 4.8, while the two score very similarly.

Anthropic’s models achieve the highest overall scores, but use substantially more output tokens. Claude Fable 5 uses about 290,000 output tokens per task on average. OpenAI’s model trajectory moves up and to the left on the chart, meaning the models are becoming both more capable and more efficient. GPT-5.6 Sol uses roughly one-tenth as many output tokens as Claude Fable 5 while ranking third overall.

Senior SWE-Bench also has a fine-grained task taxonomy, so users can inspect performance for particular stacks and task characteristics rather than relying only on an aggregate score. On tasks involving both a Python backend and a TypeScript frontend, for example, GPT-5.6 Sol leads. The taxonomy also shows model progress across releases on specific types of engineering work.

Benchmark awareness and reward hacking

One of the most interesting behavioral findings is that newer models show greater awareness that they are being benchmarked. Claude Fable 5 does or says something suggesting benchmark awareness about three times as often as Claude Opus 4.8.

That awareness can affect behavior and often corresponds with attempts at reward hacking. In a coding benchmark, a model may look on GitHub for the pull request that originally produced the task and copy the solution.

Senior SWE-Bench has controls designed to prevent that behavior, including restrictions on internet access in these evaluation runs. But models can find creative workarounds. GitHub access is blocked, for example, while agents can still install packages from PyPI and other registries. An agent might install a package associated with the repository, unpack it, locate the relevant code, and copy it.

These models are highly capable. They are trying to complete the task and achieve the reward efficiently, but that can produce unexpected behavior. Anyone building or running benchmarks should inspect trajectories closely and watch for behavior that achieves the reward without satisfying the benchmark’s intent.

Q&A

What was the most surprising discovery from the benchmark?

The benchmark-awareness behavior was not something I expected to see. It emerged late in the process of publishing the benchmark. Based on release timing, Claude Sonnet 5 was the first model we evaluated that showed substantially higher awareness.

The similarity between Sonnet 5 and Fable 5, compared with Sonnet 5 and Opus 4.8, is revealing in terms of how the models may be related.

Many existing benchmarks make assumptions based on an earlier generation of model behavior, when reward hacking was less common. A benchmark might have exploitable topology, such as the ability to find the answer on GitHub, but the vulnerability was effectively inert because models did not try to exploit it.

If people run newer models without checking for these behaviors or adding safeguards, they may report misleading results.

Which harness did you use, and how do you prevent benchmark leakage?

We used Mini-SWE-Agent as the universal harness because we wanted to benchmark model performance rather than model-plus-harness combinations. I suspect the next generation of benchmarks may move beyond that distinction toward something more holistic.

We experimented with Claude Opus using Claude Code and GPT-5.5 using Codex. Those combinations performed somewhat better, but not enough for us to report them separately here. We also did not customize the harness for individual repositories. That is an area for future improvement, although doing it at scale requires reliably identifying and using repository-specific tooling in unfamiliar codebases.

Leakage is a real concern. The Terminal-Bench team found cases where gold trajectories had been packaged into a model distribution, allowing the model to unpack and follow them step by step during evaluation.

Our main defense against leakage is a private task set. The benchmark has 100 tasks: 50 public and 50 private. The two sets were selected so current models score similarly on them. If we see a substantial divergence between public- and private-set performance, it may indicate that something unusual is happening.

How do performance and solvability vary across task types?

The task taxonomy is fully explorable on the site. You can filter tasks by languages, stacks, and other metadata, then see which models perform best for the work you care about. If you work in Elixir and Go, for example, the Claude models currently perform best, followed by GPT-5.6 Sol.

About half of the tasks currently have a zero percent tasteful solve rate. On most of those, at least one agent passes the basic behavioral checks, which indicates meaningful progress even though the solution does not meet the complete correctness and code-quality bar.

We also wanted to ensure the tasks were not fundamentally unsolvable. Each task has a guided variant that provides substantially more information about how to solve it without giving away the exact solution. All but approximately five tasks were solved in the guided setting. That helps us distinguish tasks that require more knowledge and intuition from tasks that are simply invalid.

Do you evaluate only the final code, or also the agent’s reasoning process?

We analyze the full traces. In one analysis, we divided the solving process into comprehension, investigation, design, implementation, and verification, then studied where models succeeded or failed.

Claude Opus 4.8 performs particularly well at comprehension relative to many other models, and we saw further progress from Claude Fable 5. Trace analysis helps reveal where each model is improving and where failures still occur.

What open-weight models have you evaluated?

GLM-5.2 is the most recent open-weight model currently on the leaderboard. We have also evaluated Kimi K2.6 and are running Kimi K2.7, MiniMax, and the new Thinking Machines model. We expect to add more open-weight models soon.

What were the biggest challenges in building the benchmark?

Most of our research and development effort went into making the validation-agent process highly reliable. Our early assumption was that a strong coding agent could write the needed tests from a single prompt. That did not work. The agents made mistakes in many different ways.

We had to refine the expert-provided information in the validation specification, determine which pointers mattered most, improve the revision process, and carefully design the handoff between the LLM and programmatic execution. It required much more work than we expected, but it was necessary to achieve reliable adaptive verification.

The biggest quality-control challenge is preventing materially worse solutions from passing while also preventing near-optimal alternative solutions from failing. False negatives can enter through subtle assumptions. A test helper might assume a particular API route name or payload shape even though another choice would be equally valid.

We use highly tuned LLM-based checks, run tasks repeatedly to verify consistent scoring, test multiple reference solutions, conduct multi-stage expert review, and calibrate the LLM-judge components. Our research team and software-engineering experts from Snorkel’s expert network contribute to that process.

Why focus on software infrastructure and developer tools rather than domain-specific repositories?

The benchmark includes more than infrastructure and developer tooling. It contains broader systems and full-stack applications such as Immich, an open-source photo-management application, and Paperless, a document-management system.

That said, the tasks are still situated within the software-engineering stack. Future releases will add more tasks, and domain-specific software engineering is an area we want to expand. Working in production full-stack repositories is one level of challenge; working in software that also requires deep finance, healthcare, or other domain knowledge is another.

Is there a time limit for each task?

Yes. Earlier agent benchmarks often used time limits of five or ten minutes to encourage efficient rollouts. As interest has shifted toward longer coding tasks, some benchmarks now allow six or twelve hours so agents have time to work through an entire problem.

Senior SWE-Bench uses a six-hour solver timeout, partly to control evaluation costs. Verification, including the validation agent and LLM judges, generally has a limit of roughly 10 to 15 minutes, which is more than it usually needs. The six-hour solver limit is representative of other long-horizon coding benchmarks today.

Explore Senior SWE-Bench, view the dataset on GitHub, and learn more about upcoming Snorkel AI Reading Group sessions.

Previous Reading Group sessions: Agents’ Last Exam · Olmix: Data Mixing for LLM Development · Code World Models and AutoHarness · Collaborative Gym · JudgmentBench