resources

Resource library

Explore our complete library of resources including blogs, benchmarks, research papers, and more.

Image for Data 2.0 and the research era of AI data
Blog

Data 2.0 and the research era of AI data

Announcing Snorkel’s $350M fundraise to build the frontier lab for AI data.

September 22, 2026
Image for Terminal-Bench 4.0: Why Continuous Benchmarks Require Continuous QA
Blog

Terminal-Bench 4.0: Why Continuous Benchmarks Require Continuous QA

Announcing a $3M commitment to launch Open Benchmarks Grants
August 28, 2026
Image for Grok 4.5 Testing Results: How SpaceXAI’s New Model Performs on Real Professional Work
Blog

Grok 4.5 Testing Results: How SpaceXAI’s New Model Performs on Real Professional Work

Announcing a $3M commitment to launch Open Benchmarks Grants
July 8, 2026
Image for Why coding agents need better data, evals, and environments
Blog

Why coding agents need better data, evals, and environments

Announcing a $3M commitment to launch Open Benchmarks Grants
May 11, 2026
Image for Closing the Evaluation Gap in Agentic AI
Blog

Closing the Evaluation Gap in Agentic AI

Announcing a $3M commitment to launch Open Benchmarks Grants

February 11, 2026
Image for Why Frontier Agents Fail Real Engineering Work: Two Terminal-Bench 3.0 Task Deep Dives
Blog

Why Frontier Agents Fail Real Engineering Work: Two Terminal-Bench 3.0 Task Deep Dives

Announcing a $3M commitment to launch Open Benchmarks Grants
August 24, 2026
Image for Building FinQA: An Open RL Environment for Financial Reasoning Agents
Blog

Building FinQA: An Open RL Environment for Financial Reasoning Agents

Announcing a $3M commitment to launch Open Benchmarks Grants
March 30, 2026
Image for The science of rubric design
Blog

The science of rubric design

Announcing a $3M commitment to launch Open Benchmarks Grants
September 11, 2025
Image for Benchtalks #3: We taught AI everything except how to learn
Blog

Benchtalks #3: We taught AI everything except how to learn

Featuring Parth Asawa (Continual Learning Bench)

June 25, 2026
of
Type: All Types
Sort: Newest
Osprey: Weak Supervision of Imbalanced Extraction Problems Without Code
Supervised methods are commonly used for machine-learning based applications but require expensive labeled dataset creation and maintenance. Increasingly, practitioners employ weak supervision approaches, where training labels are programmatically generated in higher-level but noisier ways. However, these approaches require domain experts with programming skills. Additionally, highly imbalanced data is often a significant practical challenge for these approaches. In this work, we propose Osprey, a weak-supervision system suited for highly imbalanced data, built on top of the Snorkel framework. In order to support non-coders, the programmatic labeling is decoupled into a code layer and a configuration one. This decoupling enables a rapid...
Research Paper
Osprey: Weak Supervision of Imbalanced Extraction Problems Without Code

Supervised methods are commonly used for machine-learning based applications but require expensive labeled dataset creation and maintenance. Increasingly, practitioners employ weak supervision approaches, where training labels are programmatically generated in higher-level but noisier ways. However, these approaches require domain experts with programming skills. Additionally, highly imbalanced data is often a significant practical challenge for these approaches. In this work, we…

Dec 12, 2019 •

Eran Bringer, Abraham Israeli, Yoav Shoham, Alex Ratner + 1 more

Learn more about Osprey: Weak Supervision of Imbalanced Extraction Problems Without Code
Multi-Resolution Weak Supervision for Sequential Data
Since manually labeling training data is slow and expensive, recent industrial and scientific research efforts have turned to weaker or noisier forms of supervision sources. However, existing weak supervision approaches fail to model multi-resolution sources for sequential data, like video, that can assign labels to individual elements or collections of elements in a sequence. A key challenge in weak supervision is estimating the unknown accuracies and correlations of these sources without using labeled data. Multi-resolution sources exacerbate this challenge due to complex correlations and sample complexity that scales in the length of the sequence. We propose Dugong, the first framework...
Research Paper
Multi-Resolution Weak Supervision for Sequential Data

Since manually labeling training data is slow and expensive, recent industrial and scientific research efforts have turned to weaker or noisier forms of supervision sources. However, existing weak supervision approaches fail to model multi-resolution sources for sequential data, like video, that can assign labels to individual elements or collections of elements in a sequence. A key challenge in weak supervision…

Dec 11, 2019 •

Frederic Sala, Paroma Varma, Jason Fries, Daniel Y. Fu + 7 more

Learn more about Multi-Resolution Weak Supervision for Sequential Data
Medical Device Surveillance With Electronic Health Records
Post-market medical device surveillance is a challenge facing manufacturers, regulatory agencies, and health care providers. Electronic health records are valuable sources of real world evidence to assess device safety and track device-related patient outcomes over time. However, distilling this evidence remains challenging, as information is fractured across clinical notes and structured records. Modern machine learning methods for machine reading promise to unlock increasingly complex information from text, but face barriers due to their reliance on large and expensive hand-labeled training sets. To address these challenges, we developed and validated state-of-the-art deep learning methods that identify patient outcomes from clinical notes...
Research Paper
Medical Device Surveillance With Electronic Health Records

Post-market medical device surveillance is a challenge facing manufacturers, regulatory agencies, and health care providers. Electronic health records are valuable sources of real world evidence to assess device safety and track device-related patient outcomes over time. However, distilling this evidence remains challenging, as information is fractured across clinical notes and structured records. Modern machine learning methods for machine reading promise…

Dec 10, 2019 •

Alison Callahan, Jason Fries, Christopher Ré, James I. Huddleston + 3 more

Learn more about Medical Device Surveillance With Electronic Health Records
Learning Dependency Structures for Weak Supervision Models
Labeling training data is a key bottleneck in the modern machine learning pipeline. Recent weak supervision approaches combine labels from multiple noisy sources by estimating their accuracies without access to ground truth labels; however, estimating the dependencies among these sources is a critical challenge. We focus on a robust PCAbased algorithm for learning these dependency structures, establish improved theoretical recovery rates, and outperform existing methods on various real-world tasks. Under certain conditions, we show that the amount of unlabeled data needed can scale sublinearly or even logarithmically with the number of sources m, improving over previous efforts that ignore the...
Research Paper
Learning Dependency Structures for Weak Supervision Models

Labeling training data is a key bottleneck in the modern machine learning pipeline. Recent weak supervision approaches combine labels from multiple noisy sources by estimating their accuracies without access to ground truth labels; however, estimating the dependencies among these sources is a critical challenge. We focus on a robust PCAbased algorithm for learning these dependency structures, establish improved theoretical recovery…

Dec 09, 2019 •
Learn more about Learning Dependency Structures for Weak Supervision Models
Interactive Programmatic Labeling for Weak Supervision
The standard supervised machine learning pipeline involves labeling individual training data points, which is often prohibitively slow and expensive. New programmatic or weak supervision approaches expedite this process by having users instead write labeling functions, simple rules or other heuristic functions that label subsets of a dataset. While these types of programmatic labeling approaches can provide significant advantages over labeling training sets by hand, there is usually little formal structure or guidance for how these labeling functions are created by users. We perform an initial exploration of processes through which users can be guided by asking them to write labeling...
Research Paper
Interactive Programmatic Labeling for Weak Supervision

The standard supervised machine learning pipeline involves labeling individual training data points, which is often prohibitively slow and expensive. New programmatic or weak supervision approaches expedite this process by having users instead write labeling functions, simple rules or other heuristic functions that label subsets of a dataset. While these types of programmatic labeling approaches can provide significant advantages over labeling…

Dec 08, 2019 •

Benjamin Cohen-Wang, Stephen Mussmann, Alex Ratner, Christopher Ré

Learn more about Interactive Programmatic Labeling for Weak Supervision
Bootstrapping Conversational Agents with Weak Supervision
Many conversational agents in the market today follow a standard bot development framework which requires training intent classifiers to recognize user input. The need to create a proper set of training examples is often the bottleneck in the development process. In many occasions agent developers have access to historical chat logs that can provide a good quantity as well as coverage of training examples. However, the cost of labeling them with tens to hundreds of intents often prohibits taking full advantage of these chat logs. In this paper, we present a framework called \textit{search, label, and propagate} (SLP) for bootstrapping...
Research Paper
Bootstrapping Conversational Agents with Weak Supervision

Many conversational agents in the market today follow a standard bot development framework which requires training intent classifiers to recognize user input. The need to create a proper set of training examples is often the bottleneck in the development process. In many occasions agent developers have access to historical chat logs that can provide a good quantity as well as…

Dec 07, 2019 •

Neil Mallinar, Abhishek Shah, Rajendra Ugrani, Ayush Gupta + 8 more

Learn more about Bootstrapping Conversational Agents with Weak Supervision
A Machine-Compiled Database of Genome-Wide Association Studies
Tens of thousands of genotype-phenotype associations have been discovered to date, yet not all of them are easily accessible to scientists. Here, we describe GWASkb, a machine-compiled knowledge base of genetic associations collected from the scientific literature using automated information extraction algorithms. Our information extraction system helps curators by automatically collecting over 6,000 associations from open-access publications with an estimated recall of 60-80% and with an estimated precision of 78-94% (measured relative to existing manually curated knowledge bases). This system represents a fully automated GWAS curation effort and is made possible by a paradigm for constructing machine learning systems called...
Research Paper
A Machine-Compiled Database of Genome-Wide Association Studies

Tens of thousands of genotype-phenotype associations have been discovered to date, yet not all of them are easily accessible to scientists. Here, we describe GWASkb, a machine-compiled knowledge base of genetic associations collected from the scientific literature using automated information extraction algorithms. Our information extraction system helps curators by automatically collecting over 6,000 associations from open-access publications with an estimated…

Dec 06, 2019 •

Volodymyr Kuleshov, Jialin Ding, Christopher Vo, Braden J. Hancock + 5 more

Learn more about A Machine-Compiled Database of Genome-Wide Association Studies
A Clinical Text Classification Paradigm Using Weak Supervision…
Background: Automatic clinical text classification is a natural language processing (NLP) technology that unlocks information embedded in clinical narratives. Machine learning approaches have been shown to be effective for clinical text classification tasks. However, a successful machine learning model usually requires extensive human efforts to create labeled training data and conduct feature engineering. In this study, we propose a clinical text classification paradigm using weak supervision and deep representation to reduce these human efforts. Methods: We develop a rule-based NLP algorithm to automatically generate labels for the training data, and then use the pre-trained word embeddings as deep representation features...
Research Paper
A Clinical Text Classification Paradigm Using Weak Supervision…

Background: Automatic clinical text classification is a natural language processing (NLP) technology that unlocks information embedded in clinical narratives. Machine learning approaches have been shown to be effective for clinical text classification tasks. However, a successful machine learning model usually requires extensive human efforts to create labeled training data and conduct feature engineering. In this study, we propose a clinical…

Dec 05, 2019 •

Yanshan Wang, Sunghwan Sohn, Sijia Liu, Feichen Shen + 4 more

Learn more about A Clinical Text Classification Paradigm Using Weak Supervision…
Training Classifiers with Natural Language Explanations
Training accurate classifiers requires many labels, but each label provides only limited information (one bit for binary classification). In this work, we propose BabbleLabble, a framework for training classifiers in which an annotator provides a natural language explanation for each labeling decision. A semantic parser converts these explanations into programmatic labeling functions that generate noisy labels for an arbitrary amount of unlabeled data, which is used to train a classifier. On three relation extraction tasks, we find that users are able to train classifiers with comparable F1 scores from 5–100× faster by providing explanations instead of just labels. Furthermore, given...
Research Paper
Training Classifiers with Natural Language Explanations

Training accurate classifiers requires many labels, but each label provides only limited information (one bit for binary classification). In this work, we propose BabbleLabble, a framework for training classifiers in which an annotator provides a natural language explanation for each labeling decision. A semantic parser converts these explanations into programmatic labeling functions that generate noisy labels for an arbitrary amount…

Dec 20, 2018 •

Braden J. Hancock, Paroma Varma, Stephanie Wang, Martin Bringmann + 2 more

Learn more about Training Classifiers with Natural Language Explanations
1 … 68 69 70 71
Image

Join our newsletter

For expert advice, the latest research, and exclusive events.
By submitting this form, I acknowledge I will receive email updates from Snorkel AI, and I agree to the Terms of Use and acknowledge that my information will be used in accordance with the Privacy Policy.