Frontier Data Summit 2026 Recap

Snorkel AI brought together the leading minds in AI benchmarking and evaluation for the Frontier Data Summit on October 8th in San Francisco. The event engaged researchers, benchmark builders, and practitioners for a full day of talks and the exchanging of ideas. Throughout the event, one main idea continued to resonate: as agents get more capable, the data and evaluations that measure them have to become a science in their own right.

Four primary themes weaved throughout the day:

• Evaluation is evolving as an empirical science. Agents are unpredictable, so they need train/test discipline, trajectory analysis and continuously maintained benchmarks.
• Benchmarks continue to require hardening. Reward hacking, contamination and broken graders are not edge cases and builders may need more of an adversarial mindset.
• The frontier is moving to open-ended work. Skill-by-skill tests are saturating, and the next benchmarks must measure insight, creativity and long-horizon work.
• Experts and agents work best together. Human judgment is most valuable at the spec, the grader and the review, with agents doing the scale.

Below, there is a short recap of each session with the full video. You can also watch the full playlist on YouTube.

Keynote: Data 2.0 and the research era of AI data

Alex Ratner opened the day by arguing that AI data has become a research and technology problem, not a volume-and-logistics one. The old unit was a crowdsourced label: cat or dog, thumbs up or thumbs down. The new unit is a task, an environment and a rubric: a simulated world where an agent attempts something hard, and a grading key that scores it. Three things follow as that complexity rises. Quality control gets harder (our complex tasks now carry high tens to low hundreds of distinct checks), the surface area for reward hacking grows, and finding the right distribution of data becomes its own problem.

He closed on evaluation. “What you can’t measure, you can’t improve,” and, to his knowledge, capabilities are outpacing our ability to measure them for the first time in AI history. Our answer is to be benchmark maximalists: if a test is too easy, build better tests, not fewer of them. That is the case for the expanded $30 million Open Benchmarks Grants program, red-teaming support, and a new research fellowship.

ARC-AGI explained: closing the human-AI gap

Snorkel chief scientist Fred Sala sat down with François Chollet, creator of Keras and founder of ARC-AGI and the ARC Prize. Chollet’s framing: benchmarks should target the residual gap between humans and AI, and co-evolve with frontier models rather than set a fixed bar. The hard question is how to score open-ended work when there is no goal to reach. His candidate answer is reuse: a real insight behaves like a compression primitive, a simple building block that simplifies or unlocks many later results. Chollet wants AI for science done in the open, with shared infrastructure for agent swarms to pass partial results, and said ARC-AGI 4, aimed at open-ended invention, is targeted for Q1 next year, with ARC-AGI 5 pushing further.

LLM evaluation needs a real science

Alex Shaw (Laude Institute, co-creator of Terminal-Bench and Harbor) and Sanmi Koyejo (Stanford Trustworthy AI Research Lab), in a session moderated by Francesca Vera, argued that agents should be studied the way machine learning is: empirically, with train and test sets, ablations and a watch for overfitting. Shaw’s one-line version: software is predictable, agents are not. Both pushed past the single score. A resolution rate tells you how much can be automated; the trajectories tell you why the rest can’t. The closing debate was how open an eval should be. Shaw’s answer was three tiers: private for internal development, shared with chosen partners, or fully public to set targets and back claims.

AI cybersecurity: when an agent cheats on a hacking exam

In a fireside chat with Alex Ratner, Dawn Song (UC Berkeley) covered her lab’s benchmarks, from Agents’ Last Exam, built to measure agents on long-horizon, economically valuable work across more than 50 sectors, to CyberGym and ExploitGym, which test the vulnerability-to-exploit pipeline. Song described a real incident in which an agent working on ExploitGym tasks hacked its way to a better score, and drew a lesson for every benchmark builder, not just security ones: build with an adversarial mindset, as if the agent were an attacker. Her group’s BenchJack work shows many benchmarks aren’t resilient to reward hacking. Alex connected this to training: if an environment rewards cheating, you are literally training models to be misaligned, which is why we’re forming a benchmark red team alongside the grants program.

Long-horizon agents: how to evaluate them without reward hacking

Moderated by Vincent Sunn Chen, this panel brought together Sudip Roy (Adaption), Niko Grupen (Harvey), Yu Su (NeoCognition and Ohio State) and Huaxiu Yao (UNC-Chapel Hill): academics building long-horizon benchmarks alongside practitioners shipping agents to customers. There was no single definition of “long horizon,” but three practical themes ran through the discussion. Experts as multipliers, not labelers: one legal AI team said giving domain experts coding agents to author synthetic environments raised environment-generation efficiency by roughly an order of magnitude, with experts still reviewing and remediating. Outcome rewards over process rewards: grade outcomes, and keep process checks to the near-certain minimum. Reward hacking is inevitable, so manage it: suggestions included chain-of-thought monitoring, comparing trajectories across models, and fitting agent work into the review hierarchies real organizations already use.

How do you evaluate an open frontier model?

Braden Hancock interviewed David Hall about Marin, a nonprofit effort at Open Athena to build a frontier-level open model and publish the whole process. The practical lesson was that evals change over a model’s life. Early in pre-training, the model can’t do anything like Terminal-Bench, so the team watches held-out loss (including a post-cutoff “Uncheatable Eval”) and uses scaling laws from small models to check the run is on track. Agentic benchmarks only become informative after mid-training and SFT, and even then “getting nonzero is pretty good.” On contamination, Hall was blunt: it is basically impossible. On the post-training side, the team has had to throw out 90 to 98 percent of some public RL datasets because of broken graders or unexecutable tasks. His call to the room: open post-training data and environments are the gap that most needs filling.

Terminal-Bench-Science: can AI agents do real science research?

Steven Dillmann (Stanford) presented Terminal-Bench-Science, a community effort from the Terminal-Bench team and scientists across institutions to evaluate agents on research workflows in the life, physical, Earth, mathematical and engineering sciences. The first release drew over 900 proposals and 386 implementations, and only 70 tasks (about 8 percent of proposals) made it in, each passing a domain reviewer, a technical reviewer and a final “bar raiser.” The results show how much headroom is left: at release the strongest model passed about 30 percent of tasks, and 13 of 17 tested models scored under 25 percent. Release 0.2 (around 100 tasks) is due toward the end of October, with 1.0 by year-end, and the team is looking for scientists to contribute. Terminal-Bench-Science received support through Snorkel’s Open Benchmarks Grants.

JudgmentBench: how lawyers actually judge AI output

Russell Yang (Microsoft) opened with a poll: is legal practice more like natural science or fine art? Most of the room said some of both, which motivated the study. JudgmentBench tests the two common ways to evaluate models in a domain where ground truth is contested: comparative judgments (which of two outputs is better) versus rubric scoring (assign points against a rubric). More than 50 expert lawyers, 63 percent with over eight years of experience, contributed 3,000 annotations on 30 tasks from Harvey’s BigLawBench. The result: comparative judgments recovered the quality ordering better, took less than half the time, and the advantage held for junior associates, senior associates and partners alike. JudgmentBench was supported by Snorkel’s Open Benchmarks Grants, and the 30 tasks and 3,000 annotations are public.

SlopCodeBench: AI coding agents write worse code over time

Gabe Orlanski (UW–Madison and MIT) argued that coding benchmarks built on one issue and one PR miss how software is really written, through iteration. SlopCodeBench starts an agent on a simple spec (a Python code search engine) and keeps changing the requirements checkpoint by checkpoint without warning, so a design that wasn’t modular eventually has to be rewritten. Across 196 checkpoints on 36 problems, one frontier model solved only 14 percent, and across the models tested only about 1 percent of checkpoints were solved on average, versus 32 percent when each task was run in isolation. The core finding is that agents never stop to refactor. In one example, a function a competent engineer would write in five lines grew to 140. Orlanski’s conclusion is that this has to be fixed in training, not only in the harness. SlopCodeBench was supported by Snorkel’s Open Benchmarks Grants.

Panel: building open AI benchmarks

The three lightning-talk speakers (Russell Yang of JudgmentBench, Gabe Orlanski of SlopCodeBench and Steven Dillmann of Terminal-Bench-Science) stayed on stage to talk shop about what it takes to build a benchmark that holds up. A few of the points that stuck:

• Verifiers must be flexible. The most common failure in submitted science tasks is a verifier tied too closely to the reference solution, so it rejects an equally valid approach. Grade outcomes, not implementation.
• Trajectories matter. Reviewers used agents to read full trajectories and reasoning traces to find ambiguous or broken tasks, and warned that the open community is losing access to those traces.
• Benchmaxing depends on the definition. Hill-climbing a well-built, diverse, continuously updated benchmark is fine; the problem is when scores stop reflecting real capability. Held-out sets, independent evaluation and simplicity were the proposed defenses.

Their advice to the next wave of builders: plan the design, annotators and analysis up front, keep the benchmark simple enough that others will actually run it, and build a team whose reviewers believe in the mission.

What’s next: Open Benchmarks Grants

Several of the benchmarks presented at the Summit were supported by our Open Benchmarks Grants, which is now expanding to a $30 million commitment. Folks interested in this space are encouraged to apply and to submit their benchmark ideas. Alongside these efforts, Snorkel AI is also expanding to build the Snorkel Fellows program, a research driven program for active academics and builders working on frontier evaluation, and Snorkel’s Open Benchmarks Red Team, an effort to help benchmark builders and teams identify and fix weaknesses in evaluations.

If you’re building a benchmark, or want to help hardening one, learn more and apply at benchmarks.snorkel.ai. You can also watch Vincent Sunn Chen’s interview with MTS on the expansion below, and browse every Summit session in the playlist.