Expo talk
The Art & Science of
Benchmarking Agents
Date
Sun, Dec 6
Time
12:00 – 1:00 PM
Location
Grand Ballroom B1
Benchmarks matter more than ever as agents take on real work, yet benchmaxxing, benchslop, and reward hacking are eroding trust in them. This talk argues that a great agent benchmark is both a research agenda (the art) and a trustworthy measurement instrument (the science), using Senior SWE-Bench as a case study. Attendees leave with a practical checklist for building agent benchmarks that stay difficult, fair, and useful.
Kiran Lakkaraju
Director of Research, Frontier Benchmarks
Accepted papers
Five papers from our research team.
Agents’ Last Exam
Long-horizon professional tasks, <1% average pass rate on the hardest tier
Snorkel authors
Amanda Dsouza, Vincent Sunn Chen
Continual Learning Bench
Do agents improve with experience?
Snorkel authors
Christopher M. Glaze, Ramya Ramakrishnan, Vincent Sunn Chen, Frederic Sala
SlopCodeBench
Agent-written code gets more bloated and eroded with every iteration
Snorkel authors
The team
The people behind the research.
Our researchers and field engineers will be at the booth throughout the conference.
Connect with us
Stay in the know
We’re planning exciting dinners and networking side events. Get notified.