We develop methods, benchmarks, and training systems that turn expert data into frontier AI
building benchmarks and collaborating with
Featured research
Senior SWE-bench built by Snorkel with Princeton University and UW-Madison
Continual Learning Bench: Evaluating agents that adapt and improve over time
Learning from Less: Measuring the Effectiveness of RLVR in Low Data Compute Regimes
Agents' Last Exam: Challenge and measure AI agents on economically valuable and real-world tasks
SlopCodeBench: A community benchmark measuring code erosion
Harvey’s BigLaw Bench: Research
Terminal-Bench 2.0: Raising the bar for AI agent evaluation
Benchmarking Agents in Insurance Underwriting Environments
Agentic Coding benchmark: Evaluating AI Models on complex, real-world coding tasks
Vision and impact
We help labs advance frontier models by working with domain experts to design and build complex, realistic datasets that drive model performance.
Benchmarking & Evaluation
Build benchmarks that define and advance the AI frontier
Scaling Subject Matter Expertise
Define how subject matter experts encode their knowledge into data
RL, Training, & Data Valuation
Drive dataset development based on feedback from RL and model training
Community and open science
Open benchmarks, conversations, and research for real-world AI performance.
Open Benchmarks Grants
Backed by a $3M commitment, the program funds open-source datasets, benchmarks, and evaluation artifacts that shape how frontier AI systems are built and evaluated.
Benchtalks
Reading Group
DEEP RESEARCH Expertise
Technical advisors and distinguished affiliates
Browse research blogs and academic papers
This paper presents Nemo, an interactive system that improves the overall productivity of Weak Supervision learning pipelines by an average of 20%, compared to the prevailing WS approach.
This paper presents a comprehensive survey of recent advances in Programmatic Weak Supervision (PWS), and discusses related approaches to tackle limited labeled data scenarios.
This paper finds that only 13% of biomedical datasets are available via programmatic access and 30% lack documentation on licensing and permitted reuse, highlighting the dataset debt in biomedical NLP.
PromptSource is a system that provides a templating language, an interface, and a set of guidelines to create, share, and use natural language prompts to train and query language models.
Amanpreet Singh, Lead Researcher at Hugging Face gave a presentation entitled Towards Unified Foundation Models for Vision and Language Alignment a Snorkel AI’s Foundation Model Summit in January.
Twelve speakers shared their insights into the present and future of foundation models January event; see what they had to say.
Foundation Models (FMs), such as GPT-3 and Stable Diffusion, mark the beginning of a new era in machine learning and artificial intelligence. What are they and how will they impact your business? Find out in our guide.
Combining foundation model outputs with weak supervision yields faster model development and requires fewer ground truth labels.
Snorkel AI CEO and Co-Founder Alex Ratner’s introduction to data-centric AI from the 2022 Future of Data-Centric AI virtual conference.
A one-day, invite-only summit providing a first look at the benchmarks and research that will shape the frontier.
