LegalBench 2: Building a benchmark for legal autograders

Nov 6, 2026

LegalBench 2 (LB2) is an ongoing project to build benchmark datasets for validating legal autograders: AI systems that grade other AI systems’ legal work. It builds on LegalBench, the NeurIPS 2023 benchmark of 162 legal reasoning tasks, and invites contributions from legal practitioners and researchers. Snorkel AI supports the project through its Open Benchmarks Grants program.


What is LegalBench 2?

Evaluating legal AI today usually requires attorneys to review model outputs by hand. That is costly, and it has to be repeated every time a model is updated.

An autograder is an AI system that grades another AI system’s output. If an autograder agrees with attorneys a substantial proportion of the time (for example, 70% or more), it could stand in for human review. Little is known, though, about how well legal autograders agree with lawyers. LB2 builds the datasets needed to measure that.

How LB2 tests an autograder

LB2 uses preference-based autograders. Each one is given a task instruction and two responses, and it picks the response an expert attorney would prefer. Its agreement rate with real attorneys’ choices measures how reliable it is. For related work on evaluating legal AI, see JudgmentBench, which compares rubric-based and preference-based evaluation.

ComponentWhat it containsExample: case summarization
InstructionsThe task promptsCases to summarize
ResponsesModel outputs for each instructionLLM-generated summaries
Human preferencesWhich of two responses an attorney prefersAttorneys choosing the better summary in each pair
Autograder scoreHow often the autograder matches the attorneys’ choicesAgreement rate, with 70%+ as an example threshold

How LB2 differs from LegalBench

LegalBench asked whether AI can perform legal tasks. LB2 asks whether AI can reliably judge the quality of AI-produced legal work.

MeasureLegalBenchLegalBench 2
QuestionCan AI perform legal tasks?Can AI reliably judge AI-produced legal work?
StatusPublished at NeurIPS 2023Ongoing project
Scope162 tasks across six types of legal reasoning, 20 models evaluatedDatasets of instructions, model responses and attorney preferences
People40 collaboratorsOpen to contributions; task-data contributors are credited as co-authors

Example task categories

The project page lists these as illustrative, not exhaustive.

  • Legal education: law school exam grading
  • Depositions: question strategy evaluation
  • Brief writing: argument quality
  • Contract drafting: clause quality
  • Regulatory analysis: compliance assessment
  • Legal research memos
  • Client advice letters
  • Due diligence: contract review
  • Demand letters
  • Sentencing advocacy: mitigation memoranda

How to contribute

Task ideas. If you can identify a legal task where AI evaluation matters, describe the task and why it matters. No data is required. Not every submission can be included.

Task data. If you also have instructions to contribute that meet minimum quality standards, they can be turned into benchmarks. Contributors who provide task data are credited as co-authors on LB2.

Contributions go through the project’s contribution form.

Snorkel AI’s role

Snorkel AI supports LegalBench 2 through its Open Benchmarks Grants program, which backs open benchmarks, red-teaming, and fellowships for open AI evaluation. See every funded benchmark.

For teams building or buying legal AI, LB2 is a way to find out whether an AI grader can be trusted to stand in for attorney review. The project is still collecting tasks and data, and no results have been published yet.

For models that need to be right. Not just good enough.