We develop methods, benchmarks, and training systems that turn expert data into frontier AI
building benchmarks and collaborating with
Featured research
Senior SWE-bench built by Snorkel with Princeton University and UW-Madison
Continual Learning Bench: Evaluating agents that adapt and improve over time
Learning from Less: Measuring the Effectiveness of RLVR in Low Data Compute Regimes
Agents' Last Exam: Challenge and measure AI agents on economically valuable and real-world tasks
SlopCodeBench: A community benchmark measuring code erosion
Harvey’s BigLaw Bench: Research
Terminal-Bench 2.0: Raising the bar for AI agent evaluation
Benchmarking Agents in Insurance Underwriting Environments
Agentic Coding benchmark: Evaluating AI Models on complex, real-world coding tasks
Vision and impact
We help labs advance frontier models by working with domain experts to design and build complex, realistic datasets that drive model performance.
Benchmarking & Evaluation
Build benchmarks that define and advance the AI frontier
Scaling Subject Matter Expertise
Define how subject matter experts encode their knowledge into data
RL, Training, & Data Valuation
Drive dataset development based on feedback from RL and model training
Community and open science
Open benchmarks, conversations, and research for real-world AI performance.
Open Benchmarks Grants
Backed by a $3M commitment, the program funds open-source datasets, benchmarks, and evaluation artifacts that shape how frontier AI systems are built and evaluated.
Benchtalks
Reading Group
DEEP RESEARCH Expertise
Technical advisors and distinguished affiliates
Browse research blogs and academic papers
Stanford assistant professor James Zou, presents “Responsible Data-Centric AI for Healthcare and Medicine” at The Future of Data-Centric AI.
Snorkel AI has accepted the first batch of applications for its first annual virtual poster competition. But there’s still time to add yours to the mix.
Join us on June 7-8 to learn how to use your data to build your AI moat at The Future of Data-Centric AI 2023 free virtual conference.
Sharon Li is an assistant professor at the University of Wisconsin-Madison. She presented “Detecting Data Distributional Shift: Challenges and Opportunities” at Snorkel AI’s The Future of Data-Centric AI Summit in 2022. The talk covered a novel approach for handling out-of-distribution objects.
Harvard Professor Vijay Janapa Reddi’s presentation: “DataPerf: Benchmarks for data” from Snorkel AI’s 2022 Future of Data-Centric AI event.
We introduce compositional soft prompting (CSP), a parameter-efficient learning technique to improve the zero-shot compositionality of large-scale pretrained vision-language models (VLMs) like CLIP. We develop CSP for compositional zero-shot learning, the task of predicting unseen attribute-object compositions (e.g., old cat and young tiger). VLMs have a flexible text encoder that can represent arbitrary classes as natural language prompts but they…
A long standing goal of the data management community is to develop general, automated systems that ingest semi-structured documents and output queryable tables without human effort or domain specific customization. Given the sheer variety of potential documents, state-of-the art systems make simplifying assumptions and use domain specific training. In this work, we ask whether we can maintain generality by using…
Prasanna Balaprakash, research and development lead from Argonne National Laboratory gave a presentation entitled “Extracting the Impact of Climate Change from Scientific Literature using Snorkel-Enabled NLP” at Snorkel AI’s Future of Data-Centric AI Workshop in August, 2022.
A one-day, invite-only summit providing a first look at the benchmarks and research that will shape the frontier.
