resources

Resource library

Explore our complete library of resources including blogs, benchmarks, research papers, and more.

Image for Data 2.0 and the research era of AI data
Blog

Data 2.0 and the research era of AI data

Announcing Snorkel’s $350M fundraise to build the frontier lab for AI data.

September 22, 2026
Image for Terminal-Bench 4.0: Why Continuous Benchmarks Require Continuous QA
Blog

Terminal-Bench 4.0: Why Continuous Benchmarks Require Continuous QA

Announcing a $3M commitment to launch Open Benchmarks Grants
August 28, 2026
Image for Grok 4.5 Testing Results: How SpaceXAI’s New Model Performs on Real Professional Work
Blog

Grok 4.5 Testing Results: How SpaceXAI’s New Model Performs on Real Professional Work

Announcing a $3M commitment to launch Open Benchmarks Grants
July 8, 2026
Image for Why coding agents need better data, evals, and environments
Blog

Why coding agents need better data, evals, and environments

Announcing a $3M commitment to launch Open Benchmarks Grants
May 11, 2026
Image for Closing the Evaluation Gap in Agentic AI
Blog

Closing the Evaluation Gap in Agentic AI

Announcing a $3M commitment to launch Open Benchmarks Grants

February 11, 2026
Image for Why Frontier Agents Fail Real Engineering Work: Two Terminal-Bench 3.0 Task Deep Dives
Blog

Why Frontier Agents Fail Real Engineering Work: Two Terminal-Bench 3.0 Task Deep Dives

Announcing a $3M commitment to launch Open Benchmarks Grants
August 24, 2026
Image for Building FinQA: An Open RL Environment for Financial Reasoning Agents
Blog

Building FinQA: An Open RL Environment for Financial Reasoning Agents

Announcing a $3M commitment to launch Open Benchmarks Grants
March 30, 2026
Image for The science of rubric design
Blog

The science of rubric design

Announcing a $3M commitment to launch Open Benchmarks Grants
September 11, 2025
Image for Benchtalks #3: We taught AI everything except how to learn
Blog

Benchtalks #3: We taught AI everything except how to learn

Featuring Parth Asawa (Continual Learning Bench)

June 25, 2026
of
Type: All Types
Sort: Newest
Software 2.0 and Snorkel: Beyond Hand-Labeled Data
In the last few years, deep learning models have simultaneously achieved high quality on conventionally challenging tasks and become easy-to-use commodity tools. These factors, combined with the ease of deployment compared to traditional software, have led to deep learning models replacing production software stacks in not only traditional machine learning-driven products including translation and search, but also in many previously heuristic-based applications. This new mode of software construction and deployment has been called Software 2.0 [2]. A key bottleneck in the construction of Software 2.0 applications is the need for large, high-quality training sets for each task. This talk describes...
Research Paper
Software 2.0 and Snorkel: Beyond Hand-Labeled Data

In the last few years, deep learning models have simultaneously achieved high quality on conventionally challenging tasks and become easy-to-use commodity tools. These factors, combined with the ease of deployment compared to traditional software, have led to deep learning models replacing production software stacks in not only traditional machine learning-driven products including translation and search, but also in many previously…

Dec 19, 2018 •
Learn more about Software 2.0 and Snorkel: Beyond Hand-Labeled Data
Snorkel MeTaL: Weak Supervision for Multi-Task Learning
Many real-world machine learning problems are challenging to tackle for two reasons: (i) they involve multiple sub-tasks at different levels of granularity; and (ii) they require large volumes of labeled training data. We propose Snorkel MeTaL, an end-to-end system for multi-task learning that leverages weak supervision provided at multiple levels of granularity by domain expert users. In MeTaL, a user specifies a problem consisting of multiple, hierarchically-related sub-tasks-for example, classifying a document at multiple levels of granularity-and then provides labeling functions for each sub-task as weak supervision. MeTaL learns a re-weighted model of these labeling functions, and uses the combined...
Research Paper
Snorkel MeTaL: Weak Supervision for Multi-Task Learning

Many real-world machine learning problems are challenging to tackle for two reasons: (i) they involve multiple sub-tasks at different levels of granularity; and (ii) they require large volumes of labeled training data. We propose Snorkel MeTaL, an end-to-end system for multi-task learning that leverages weak supervision provided at multiple levels of granularity by domain expert users. In MeTaL, a user…

Dec 18, 2018 •

Alex Ratner, Braden J. Hancock, Jared Dunnmon, Roger Goldman + 1 more

Learn more about Snorkel MeTaL: Weak Supervision for Multi-Task Learning
Fonduer: Knowledge Base Construction From Richly Formatted Data
We focus on knowledge base construction (KBC) from richly formatted data. In contrast to KBC from text or tabular data, KBC from richly formatted data aims to extract relations conveyed jointly via textual, structural, tabular, and visual expressions. We introduce Fonduer, a machine-learning-based KBC system for richly formatted data. Fonduer presents a new data model that accounts for three challenging characteristics of richly formatted data: (1) prevalent document-level relations, (2) multimodality, and (3) data variety. Fonduer uses a new deep-learning model to automatically capture the representation (i.e., features) needed to learn how to extract relations from richly formatted data. Finally,...
Research Paper
Fonduer: Knowledge Base Construction From Richly Formatted Data

We focus on knowledge base construction (KBC) from richly formatted data. In contrast to KBC from text or tabular data, KBC from richly formatted data aims to extract relations conveyed jointly via textual, structural, tabular, and visual expressions. We introduce Fonduer, a machine-learning-based KBC system for richly formatted data. Fonduer presents a new data model that accounts for three challenging…

Dec 17, 2018 •

Sen Wu, Luke Hsiao, Xiao Cheng, Braden J. Hancock + 3 more

Learn more about Fonduer: Knowledge Base Construction From Richly Formatted Data
Deep Text Mining of Instagram Data Without Strong Supervision
With the advent of social media, our online feeds increasingly consist of short, informal, and unstructured text. This textual data can be analyzed for the purpose of improving user recommendations and detecting trends. Instagram is one of the largest social media platforms, containing both text and images. However, most of the prior research on text processing in social media is focused on analyzing Twitter data, and little attention has been paid to text mining of Instagram data. Moreover, many text mining methods rely on annotated training data, which in practice is both difficult and expensive to obtain. In this paper,...
Research Paper
Deep Text Mining of Instagram Data Without Strong Supervision

With the advent of social media, our online feeds increasingly consist of short, informal, and unstructured text. This textual data can be analyzed for the purpose of improving user recommendations and detecting trends. Instagram is one of the largest social media platforms, containing both text and images. However, most of the prior research on text processing in social media is…

Dec 16, 2018 •

Kim Hammar, Shatha Jaradat, Nima Dokoohaki, Mihhail Matskin

Learn more about Deep Text Mining of Instagram Data Without Strong Supervision
Snorkel: Fast Training Set Generation for Information Extraction
State-of-the art machine learning methods such as deep learning rely on large sets of hand-labeled training data. Collecting training data is prohibitively slow and expensive, especially when technical domain expertise is required; even the largest technology companies struggle with this challenge. We address this critical bottleneck with Snorkel, a new system for quickly creating, managing, and modeling training sets. Snorkel enables users to generate large volumes of training data by writing labeling functions, which are simple functions that express heuristics and other weak supervision strategies. These user-authored labeling functions may have low accuracies and may overlap and conflict, but Snorkel...
Research Paper
Snorkel: Fast Training Set Generation for Information Extraction

State-of-the art machine learning methods such as deep learning rely on large sets of hand-labeled training data. Collecting training data is prohibitively slow and expensive, especially when technical domain expertise is required; even the largest technology companies struggle with this challenge. We address this critical bottleneck with Snorkel, a new system for quickly creating, managing, and modeling training sets. Snorkel…

Learn more about Snorkel: Fast Training Set Generation for Information Extraction
Learning to Compose Domain-Specific Transformations for Data Augmentation
Data augmentation is a ubiquitous technique for increasing the size of labeled training sets by leveraging task-specific data transformations that preserve class labels. While it is often easy for domain experts to specify individual transformations, constructing and tuning the more sophisticated compositions typically needed to achieve state-of-the-art results is a time-consuming manual task in practice. We propose a method for automating this process by learning a generative sequence model over user-specified transformation functions using a generative adversarial approach. Our method can make use of arbitrary, non-deterministic transformation functions, is robust to misspecified user input, and is trained on unlabeled data....
Research Paper
Learning to Compose Domain-Specific Transformations for Data Augmentation

Data augmentation is a ubiquitous technique for increasing the size of labeled training sets by leveraging task-specific data transformations that preserve class labels. While it is often easy for domain experts to specify individual transformations, constructing and tuning the more sophisticated compositions typically needed to achieve state-of-the-art results is a time-consuming manual task in practice. We propose a method for…

Dec 19, 2017 •

Alex Ratner, Henry Ehrenberg, Zeshan Hussain, Jared Dunnmon + 1 more

Learn more about Learning to Compose Domain-Specific Transformations for Data Augmentation
Learning the Structure of Generative Models Without Labeled Data
Curating labeled training data has become the primary bottleneck in machine learning. Recent frameworks address this bottleneck with generative models to synthesize labels at scale from weak supervision sources. The generative model's dependency structure directly affects the quality of the estimated labels, but selecting a structure automatically without any labeled data is a distinct challenge. We propose a structure estimation method that maximizes the $\ell_1$-regularized marginal pseudolikelihood of the observed data. Our analysis shows that the amount of unlabeled data required to identify the true structure scales sublinearly in the number of possible dependencies for a broad class of models....
Research Paper
Learning the Structure of Generative Models Without Labeled Data

Curating labeled training data has become the primary bottleneck in machine learning. Recent frameworks address this bottleneck with generative models to synthesize labels at scale from weak supervision sources. The generative model’s dependency structure directly affects the quality of the estimated labels, but selecting a structure automatically without any labeled data is a distinct challenge. We propose a structure estimation…

Dec 18, 2017 •
Learn more about Learning the Structure of Generative Models Without Labeled Data
Inferring Generative Model Structure With Static Analysis
Obtaining enough labeled data to robustly train complex discriminative models is a major bottleneck in the machine learning pipeline. A popular solution is combining multiple sources of weak supervision using generative models. The structure of these models affects training label quality, but is difficult to learn without any ground truth labels. We instead rely on these weak supervision sources having some structure by virtue of being encoded programmatically. We present Coral, a paradigm that infers generative model structure by statically analyzing the code for these heuristics, thus reducing the data required to learn structure significantly. We prove that Coral's sample...
Research Paper
Inferring Generative Model Structure With Static Analysis

Obtaining enough labeled data to robustly train complex discriminative models is a major bottleneck in the machine learning pipeline. A popular solution is combining multiple sources of weak supervision using generative models. The structure of these models affects training label quality, but is difficult to learn without any ground truth labels. We instead rely on these weak supervision sources having…

Dec 17, 2017 •

Paroma Varma, Bryan He, Payal Bajaj, Imon Banerjee + 3 more

Learn more about Inferring Generative Model Structure With Static Analysis
Swellshark: A Generative Model for Biomedical Named Entity Recognition Without Labeled Data
We present SwellShark, a framework for building biomedical named entity recognition (NER) systems quickly and without hand-labeled data. Our approach views biomedical resources like lexicons as function primitives for autogenerating weak supervision. We then use a generative model to unify and denoise this supervision and construct large-scale, probabilistically labeled datasets for training high-accuracy NER taggers. In three biomedical NER tasks, SwellShark achieves competitive scores with state-of-the-art supervised benchmarks using no hand-labeled training data. In a drug name extraction task using patient medical records, one domain expert using SwellShark achieved within 5.1% of a crowdsourced annotation approach -- which originally utilized...
Research Paper
Swellshark: A Generative Model for Biomedical Named Entity Recognition Without Labeled Data

We present SwellShark, a framework for building biomedical named entity recognition (NER) systems quickly and without hand-labeled data. Our approach views biomedical resources like lexicons as function primitives for autogenerating weak supervision. We then use a generative model to unify and denoise this supervision and construct large-scale, probabilistically labeled datasets for training high-accuracy NER taggers. In three biomedical NER tasks,…

Nov 13, 2017 •
Learn more about Swellshark: A Generative Model for Biomedical Named Entity Recognition Without Labeled Data
1 … 69 70 71
Image

Join our newsletter

For expert advice, the latest research, and exclusive events.
By submitting this form, I acknowledge I will receive email updates from Snorkel AI, and I agree to the Terms of Use and acknowledge that my information will be used in accordance with the Privacy Policy.