PaperBench: Evaluating AI’s Ability to Replicate AI Research
OpenAI’s PaperBench tests whether AI agents can replicate 20 ICML 2024 Spotlight and Oral papers from scratch, not summarize them, rebuild them. Snorkel AI’s PaperBench+ dataset extends the same idea with harder, original, expert-authored replication tasks.
20
ICML 2024 Spotlight &
Oral papers agents
must replicate
end to end
8,316
Individually gradable
rubric tasks,
co-developed with
the papers’ own authors
21.0%
Best tested agent’s
average replication
score — well behind
the human baseline
What is PaperBench?
PaperBench is a benchmark from OpenAI evaluating whether AI agents can replicate state-of-the-art AI research. Agents are given 20 ICML 2024 Spotlight and Oral papers and asked to reproduce each one from scratch: understand the paper’s contributions, write a working codebase, and successfully execute the experiments the paper describes. Nothing is handed over except the paper itself.
Grading a research replication is not a simple pass/fail check, so OpenAI built hierarchical rubrics that decompose each paper into smaller, objectively gradable sub-tasks — 8,316 of them in total across the 20 papers. Every rubric was co-developed with the paper’s own authors to keep it accurate and realistic. Because grading at that scale by hand isn’t practical, OpenAI also trained an LLM-based judge to score replication attempts automatically, and validated the judge’s reliability against a separate benchmark built just to test judges.
Why PaperBench matters
Most coding and reasoning benchmarks reward agents for producing an answer that looks right. PaperBench asks for something much harder to fake: a working system that actually reproduces a paper’s claimed results. That combination — reading dense scientific text, forming a correct mental model of a method, writing real code, and running it to a correct outcome — is a reasonable proxy for whether an AI system could meaningfully accelerate AI research itself, one of the more consequential capability thresholds frontier labs are watching for.
It also gives the field a concrete, falsifiable answer to “can AI do research yet?” instead of an anecdote. As later results show, the answer in 2025 was clearly not yet, which is exactly the kind of finding a good benchmark is supposed to produce.
Key findings
- The best agent scored 21.0%. Across every frontier model OpenAI tested — including GPT-4o, o1, o3-mini, DeepSeek-R1, Claude 3.5 Sonnet (New), and Gemini 2.0 Flash — the strongest performer was Claude 3.5 Sonnet (New) running with open-source agent scaffolding, averaging a 21.0% replication score across the 20 papers.
- Humans are still well ahead. OpenAI recruited top ML PhD researchers to attempt a subset of PaperBench under time pressure. Given 48 hours per paper, the human baseline averaged roughly double the best agent’s score, models do not yet outperform human experts at this task.
- Understanding isn’t the same as executing. Agents were often able to describe what a paper’s method does, but consistently lost points on the parts of the rubric that required a fully working, correctly configured codebase producing the right experimental result, a comprehension-versus-execution gap that shows up across most agentic benchmarks, including Snorkel’s own computer-use research (see Cua-Bench).
- The benchmark is open. OpenAI open-sourced the PaperBench code so other teams can reproduce the evaluation and build on it.
Snorkel AI’s role
Snorkel AI publishes PaperBench+, an expert-extended edition of PaperBench inside the Snorkel Data Series. Rather than reusing the original 20 papers, PaperBench+ is built from original, expert-authored research-replication tasks constructed in the same spirit, then hardened: longer horizons, multi-paper and multi-skill tasks, richer metadata, and difficulty calibrated against current frontier models rather than the 2024 cohort PaperBench was built around.
That calibration matters because agent capability keeps moving. As frontier models close in on the original PaperBench’s 20-paper set, a fixed public benchmark stops discriminating between good and great systems. PaperBench+ exists to keep giving frontier labs a benchmark with headroom.
