We develop methods, benchmarks, and training systems that turn expert data into frontier AI
building benchmarks and collaborating with
Featured research
Senior SWE-bench built by Snorkel with Princeton University and UW-Madison
Continual Learning Bench: Evaluating agents that adapt and improve over time
Learning from Less: Measuring the Effectiveness of RLVR in Low Data Compute Regimes
Agents' Last Exam: Challenge and measure AI agents on economically valuable and real-world tasks
SlopCodeBench: A community benchmark measuring code erosion
Harvey’s BigLaw Bench: Research
Terminal-Bench 2.0: Raising the bar for AI agent evaluation
Benchmarking Agents in Insurance Underwriting Environments
Agentic Coding benchmark: Evaluating AI Models on complex, real-world coding tasks
Vision and impact
We help labs advance frontier models by working with domain experts to design and build complex, realistic datasets that drive model performance.
Benchmarking & Evaluation
Build benchmarks that define and advance the AI frontier
Scaling Subject Matter Expertise
Define how subject matter experts encode their knowledge into data
RL, Training, & Data Valuation
Drive dataset development based on feedback from RL and model training
Community and open science
Open benchmarks, conversations, and research for real-world AI performance.
Open Benchmarks Grants
Backed by a $3M commitment, the program funds open-source datasets, benchmarks, and evaluation artifacts that shape how frontier AI systems are built and evaluated.
Benchtalks
Reading Group
DEEP RESEARCH Expertise
Technical advisors and distinguished affiliates
Browse research blogs and academic papers
Widespread adoption of electronic health records (EHRs) has fueled the development of using machine learning to build prediction models for various clinical outcomes. However, this process is often constrained by having a relatively small number of patient records for training the model. We demonstrate that using patient representation schemes inspired from techniques in natural language processing can increase the accuracy…
This work demonstrates how organizational resources, in the form of aggregate statistics, knowledge bases, and existing services can be used to connect new and existing data modalities.
Electrophysiological signals exhibit both periodic and aperiodic properties. Periodic oscillations have been linked to numerous physiological, cognitive, behavioral and disease states. Emerging evidence demonstrates that the aperiodic component has putative physiological interpretations and that it dynamically changes with age, task demands and cognitive states. Electrophysiological neural activity is typically analyzed using canonically defined frequency bands, without consideration of the aperiodic…
Introducing FlyingSquid, a weak supervision framework that runs orders of magnitude faster than previous weak supervision approaches and requires fewer assumptions
This paper proposes cross-modal data programming (XMDP) for machine learning (ML) in medicine.
This paper provides a series of results studying how performance scales with changes in source coverage, source accuracy, and the Lipschitzness of label distributions in the embedding space, and compare this rate to standard weak supervision.
Presenting Trove, a framework for weakly supervised entity classification using medical ontologies and expert-generated rules.
Background: The aortic valve is an important determinant of cardiovascular physiology and anatomic location of common human diseases. Methods: From a sample of 34 287 white British ancestry participants, we estimated functional aortic valve area by planimetry from prospectively obtained cardiac magnetic resonance imaging sequences of the aortic valve. Aortic valve area measurements were submitted to genome-wide association testing, followed by…
In this paper, we propose a learning algorithm for training deep neural networks when there is not sufficient labeled data. To improve the generalization capabilities of the deep model, we adopt a learning scheme to train two related tasks simultaneously. One is the original task (target), and the other is an auxiliary task (source). In order to create a related…
A one-day, invite-only summit providing a first look at the benchmarks and research that will shape the frontier.
