Senior SWE-bench

Senior SWE-Bench evaluates coding agents on work that resembles assignments given to experienced software engineers. Its initial release contains 100 tasks, split between 50 public and 50 private tasks, across 12 open-source repositories. Agents must build features from realistic instructions, investigate difficult bugs and performance issues, and produce changes that fit the surrounding codebase.

Senior SWE-Bench covers features, migrations, bugs, and performance problems across libraries, developer tools, services, and full applications. The repositories use technology stacks including Python, TypeScript, Go, Rust, Elixir, and SQL. Feature tasks may touch several services or data layers, while investigation tasks can require agents to start applications, inspect logs, reproduce failures, profile performance, and identify a root cause.

The tasks are derived from recent pull requests merged into real open-source projects. Most source pull requests were authored by engineers with at least 100 commits in the relevant repository, with maintainer-authored work receiving additional priority. This provides an authoritative reference implementation and helps ensure that each task represents work that was genuinely needed by the project.

The private half of the benchmark limits exposure to the complete evaluation set. This reduces the risk that models or agent systems can improve their leaderboard results by training directly on every available task.

Real engineering requests rarely provide the complete solution structure, exact interfaces, affected files, and tests. Senior SWE-Bench gives agents messages closer to what they might receive from a product manager, teammate, or production user. The agent must inspect the repository and infer the implementation details from the requested behaviour and existing codebase practices.

This tests whether the agent can exercise technical judgment. A coding system may perform well when every required change is stated explicitly, then struggle when it must decide where the change belongs or how the surrounding system constrains the solution.

Senior SWE-Bench combines runtime testing with task-specific and code-quality evaluation. Pre-written verifiers check required behaviour and regressions. For Design-and-Build tasks, a validation agent can generate additional behavioural tests that match the interface chosen by the solving agent. Task rubrics cover requirements that are difficult to verify through execution alone.

The evaluation also examines patch size, alignment with repository practices, and code quality relative to an expert reference solution. This allows the benchmark to accept different valid implementations while still rejecting solutions that work through fragile shortcuts or poor engineering choices.

Basic Solve Rate measures the share of attempts that pass every required pre-written verifier and automated validation test. It establishes that the submitted change works according to the benchmark’s runtime checks.

Tasteful Solve Rate adds four quality gates. The task rubric must exceed its threshold, the patch must remain below twice the reference solution’s source-line count, codebase-practice alignment must score above 2 out of 5, and relative code quality must also score above 2 out of 5. An attempt counts as tasteful only after it passes the functional tests and every additional gate.

The validation agent receives an expert-authored specification describing which behaviours to test and how to test them. It first inspects the coding agent’s patch to understand the chosen implementation, then writes system-level tests adapted to that solution. Those tests are executed through the repository’s normal test tooling.

A separate judge reviews the generated tests and their execution for fidelity to the validation specification. It can accept the tests, request one revision, or discard the validation trial when the test-writing agent has diverged from its assigned role. This process lets Senior SWE-Bench evaluate open-ended feature designs through executable behaviour rather than requiring every agent to implement the same interface.

SWE-Bench primarily asks agents to resolve GitHub issues using repository code and a fixed test suite. Senior SWE-Bench places more emphasis on open-ended feature design, runtime investigation, unstated codebase requirements, and the quality of the resulting patch.

Senior SWE-Bench instructions are also substantially less prescriptive than typical SWE-Bench Pro tasks. Its feature requests resemble product requirements, while its bug tasks resemble behavioural reports. Adaptive validation allows different valid implementation designs to be tested without revealing a detailed solution specification in advance.

The 50 public tasks are available as an open-source, Harbor-compatible dataset. Researchers can run the benchmark directly from Harbor Hub or the GitHub repository by selecting their model and agent harness. Each task supplies a sandboxed repository environment and the evaluation components needed to test the resulting patch.