Frequently Asked Questions

Snorkel AI is the frontier AI data lab. We build the data and environments behind advanced AI: the datasets, benchmarks, evaluations, and custom agents that help frontier and agentic systems work in the real world. We were founded out of the Stanford AI Lab in 2019, on a simple idea. Better data makes better AI.

Three things that feed each other. We build a platform for developing data and environments. We run as a research engine, benchmarking and publishing what actually makes data high-performing. And we deliver outcomes through embedded collaboration, co-developing datasets, benchmarks, and custom agents alongside our customers.

Leading frontier AI labs and enterprises working on specialized, high-consequence problems. Frontier teams come to us for benchmark creation and RL/agentic evaluation. Enterprises come to build reliable, domain-specific AI grounded in their own data and operating knowledge.

In high-stakes, specialized work, it’s quality that determines the last 1-2% of accuracy that unlocks deployment, not volume. So we define quality in measurable, defensible terms: calibrated expert signal, clear rubrics and verifiers, adjudication, provenance, and coverage of the edge cases.

No. Snorkel today is a frontier AI data lab, not a self-serve labeling tool. We deliver expert data development as a service: datasets, benchmarks, evaluations, and environments built with our experts, rather than a SaaS app your team runs. If you need labeled or annotated data for a specialized, high-stakes use case, we can almost certainly help though. Talk to a data researcher.

Yes. Whether it’s a benchmark, an eval set, RL environments, or specialized training data, we can help on essentially any data or evaluation project in a high-stakes domain. Start with a short scoping conversation, and we’ll recommend an off-the-shelf product, a custom build, or a mix.

Snorkel runs as a research engine. We experiment, benchmark, and publish on what makes data high-performing for frontier and agentic AI, then apply those findings in customer data and environments. It’s a dual motion: we develop frontier data and we deploy agentic systems, and each one sharpens the other. The methodology is the product, not just the labor.

Through research-validated methods and a steady feedback loop with leading AI labs. We benchmark data approaches, run RL and evaluation experiments, and use the results to refine task design, difficulty calibration, and verification.

Snorkel is a research-driven data lab, not a volume shop. The difference is measurable quality: calibrated expert signal, rubrics and verifiers, difficulty calibration against frontier models, contamination controls, and a feedback loop with leading AI labs, all aimed at the hardest specialized problems.

Snorkel pairs a purpose-built platform and dedicated expert and forward-deployed teams with a research-validated methodology. That mix of data development, evaluation tooling, and embedded delivery gets you to reliable, measurable results faster, and with more defensible quality, than building the same infrastructure yourself.

DaaS helps frontier AI teams build the data and environments they need for domain-specific, high-consequence problems. It comes two ways: ready-to-use products and focused custom data development. That lets teams move quickly now while building a foundation for new benchmarks and evals as priorities shift.

The Snorkel Data Series is a set of research-defined, non-exclusive dataset and environment products, refreshed quarterly as models advance. Each one is quality-validated, documented, and customizable. You get immediate value plus a foundation you can extend.

It depends on how mature and specific your use case is. Many teams start with a Snorkel Data Series product for speed, then extend it with custom development for their domain. Frontier teams often run both at once.

Both. Snorkel provides the data and the environments you need to develop and evaluate against it: reproducible execution harnesses, verifiers, and reward modules, not just static files.

Yes. The Snorkel Data Series gives you ready-to-use, expert-authored datasets and agentic environments: research-defined, quality-validated, and refreshed quarterly. You can start training and evaluating right away, without commissioning a custom build first.

Yes. We publish expert-extended “+” editions of the benchmarks frontier teams care about: Terminal-Bench+ for agentic coding and terminal tasks, SWE-Bench+ for repo-grounded software engineering, Enterprise Agentic Environments for tau2-bench-style policy-and-tool workflows, CUA-Bench+ for computer-use tasks, plus GDPval+ and PaperBench+. Each adds harder, original tasks and stronger verification.

No. They’re original, expert-authored tasks built in the spirit of those benchmarks, then hardened: longer horizons, multi-skill tasks, richer metadata, and frontier-calibrated difficulty. Terminal-Bench+ stays compatible with the Harbor Terminal-Bench format, so it drops into your existing pipelines.

Tasks are tiered by how well current frontier agents do on them, then organized into a deliberate spread from basic to frontier-difficulty. You can buy by tier, including a frontier subset where even leading models pass only a minority of attempts.

Diversity is engineered, not assumed. We look past surface semantic similarity to the skills, tools, and languages a task actually exercises, run similarity checks to catch clustering, and rebalance skewed category distributions. For coding datasets, that means spreading coverage across many programming languages instead of over-indexing on one.

Through originality validation against public datasets, systematic filtering with rejection criteria, and provenance tracking. The result: tasks haven’t leaked into pretraining, so evaluation stays meaningful and training signal stays clean.

Several, and often combined: deterministic unit tests, rubric-based verifiers, LLM-as-a-judge scoring, preference and reward-model data, and milestone-based rewards. Together they cover both outcome-level and process-level signals for agentic workflows.

Either works. Snorkel can supply the full verification layer of tests, rubrics, and reward modules. Or we can deliver tasks and reference solutions for environments where your team builds and owns the verifier. We design to your contracted spec.

Yes. The environments are built for the full lifecycle: SFT trajectory generation, RL with verifiable rewards (tests, rubrics, LLM-as-judge), and evaluation. A single asset supports post-training and benchmarking.

No. Tasks are defined by clear requirements and success criteria, not a prescribed solution path. They admit multiple valid approaches and won’t lock you into a particular agent architecture.

Yes, when that’s the goal: containerized desktop environments, our CUA-Bench+ line, with ordered action traces grounded in real applications. For engineering datasets where customers don’t want click-and-keystroke solving, we define tasks by requirements instead and verify on the output.

Yes. Snorkel builds datasets and environments for narrow, expert domains: mechanical and CAD/3D design, scientific reasoning, and regulated workflows, among others. We produce the output formats those domains require and validate the work with credentialed experts.

Yes. Sample data is available for Snorkel Data Series products, so your technical team can inspect task design, formats, and verification before scoping a purchase or custom build.

Collaboratively. We align on a task taxonomy and concrete examples of what “good” looks like, then design the dataset around them. We iterate on samples so the spec matches your training or evaluation intent before anything scales.

Against expert-defined, task-specific criteria, not generic leaderboards. Rubrics score both the final output and the process: correct tool use, retrieval quality, and how the model handles missing or uncertain information. The metrics map to your real success criteria.

Yes, it’s a core offering. We define the task taxonomy, source or generate tasks, write rubrics, calibrate difficulty, and validate coverage, across both static datasets and interactive environments.

Yes. With adversarial red teaming, failure-taxonomy design, and frontier-difficulty subsets calibrated to where leading models break, we concentrate data and evaluation on the exact weaknesses you want to close.

Yes. Snorkel Solutions builds custom agents for specialized, high-impact enterprise workflows where off-the-shelf LLMs and vertical tools fall short. We combine agent development, evaluation, and tuning with Snorkel’s data technology and expert workflows.

Reliable agents take more than prompting and orchestration. We develop the benchmarks, evaluations, and training data needed to measure performance, tune behavior, and improve quality over time, through a continuous Evaluate → Curate → Refine loop.

Everything from scoping the right high-value use case through development, deployment in your environment, monitoring, and iteration. The goal is production-ready systems that work in real operating environments, not just demos.

Yes, we deliver tiered quality. High-standard “golden” trajectories and rubric-graded sets serve evaluation and post-training. Larger-volume, high-diversity data suits pre-training. You pick the quality-and-volume mix for each use case.

Yes. Depending on the product and tier, one program can supply pre-training-scale volume, post-training data for SFT and RL, and clean evaluation sets.

Programs scale into the tens of thousands of tasks. Off-the-shelf products carry the best pricing. Volume can be accelerated with advance notice, so forecasting demand early helps you avoid rush surcharges.

Yes. Many teams buy in-demand Snorkel Data Series products against a standing budget, then review the catalog on a recurring cadence to pick what’s most useful. It’s a practical way to build a reusable data library for future work.

Yes. Snorkel works across text, code, image, audio, and video, and builds long-context and vision-language (VLM) datasets and benchmarks for multimodal and long-horizon reasoning.

Yes. As a research lab in constant contact with leading teams, we share non-confidential insights on which data and benchmark approaches are working best. We never expose any customer’s proprietary work.

Calibrated expert signal, backed by layered controls: clear guidelines, calibration sessions, multi-stage and dual-expert review, consensus mechanisms, statistical sampling, provenance, and audit trails. Programmatic quality control runs alongside expert review at every stage.

Domain experts with verified credentials: PhDs, graduate degrees, and professional certifications. They work alongside scaled teams, with coverage across many specialized sub-domains for nuanced, expert-level judgment.

Specialized, high-consequence domains: software engineering, scientific and STEM reasoning, finance, insurance underwriting, healthcare and clinical, legal and regulatory compliance, manufacturing and engineering, and government.

Yes. Snorkel builds medical reasoning datasets and agentic clinical environments. That includes EHR-grounded tasks on FHIR-formatted records where the model has to actively retrieve information rather than read a pre-supplied vignette. Evaluation uses clinician-authored rubrics, deterministic checks on structured outputs like medications and dosages (not just LLM-as-a-judge), handling of negative findings, and grounding in clinical guidelines.

Yes. Credentialed legal experts, including JD-qualified reviewers, assess assistant outputs against rubrics for usefulness to a lawyer, explainability, and verifiability, with citation checking included. A multi-reviewer calibration process surfaces disagreements and sets thresholds, so subjective quality gets measured consistently.

Yes. Snorkel builds enterprise environments and evaluations for underwriting and finance workflows: multi-agent architectures, tool use, and proprietary calculations. Custom evaluators run in CI pipelines, and SME-built golden data keeps quality tracked over time.

Snorkel maintains SOC 2 Type 2 compliance.

Yes. Snorkel is built to work on proprietary, sensitive, and regulated data, including controlled information in government settings. Data access, handling, and security controls are scoped at the start of every engagement.

Yes. Snorkel is model-agnostic and routinely compares frontier and open models to find the best fit for a workload, weighing performance against cost. It works alongside major cloud and data platforms and your existing model endpoints.

Engagements usually start with a scoped discovery phase covering use case, success criteria, and data access, then move to a pilot or initial build, then a staged rollout. Off-the-shelf Snorkel Data Series products let you start faster.

Some involvement is expected: defining rubrics, reviewing data, and acceptance testing. Pinning that down early keeps timelines on track, and Snorkel supplements with its own credentialed experts to lighten the load on your team.

Start with a scoped discovery conversation about your use case, data, and success criteria. From there, Snorkel will recommend whether to begin with a Snorkel Data Series product, a custom data build, or a custom agent engagement. Talk to a data researcher.