LegalBench 2 (LB2) is an ongoing project to build benchmark datasets for validating legal autograders: AI systems that grade other AI systems’ legal work. It builds on LegalBench, the NeurIPS 2023 benchmark of 162 legal reasoning tasks, and invites contributions from legal practitioners and researchers. Snorkel AI supports the project through its Open Benchmarks Grants program.
What is LegalBench 2?
Evaluating legal AI today usually requires attorneys to review model outputs by hand. That is costly, and it has to be repeated every time a model is updated.
An autograder is an AI system that grades another AI system’s output. If an autograder agrees with attorneys a substantial proportion of the time (for example, 70% or more), it could stand in for human review. Little is known, though, about how well legal autograders agree with lawyers. LB2 builds the datasets needed to measure that.
162
Tasks in the original LegalBench
6
Types of legal reasoning it covered
20
AI models evaluated in LegalBench
70%+
Example attorney agreement for a useful autograder
How LB2 tests an autograder
LB2 uses preference-based autograders. Each one is given a task instruction and two responses, and it picks the response an expert attorney would prefer. Its agreement rate with real attorneys’ choices measures how reliable it is. For related work on evaluating legal AI, see JudgmentBench, which compares rubric-based and preference-based evaluation.
| Component | What it contains | Example: case summarization |
|---|---|---|
| Instructions | The task prompts | Cases to summarize |
| Responses | Model outputs for each instruction | LLM-generated summaries |
| Human preferences | Which of two responses an attorney prefers | Attorneys choosing the better summary in each pair |
| Autograder score | How often the autograder matches the attorneys’ choices | Agreement rate, with 70%+ as an example threshold |
How LB2 differs from LegalBench
LegalBench asked whether AI can perform legal tasks. LB2 asks whether AI can reliably judge the quality of AI-produced legal work.
| Measure | LegalBench | LegalBench 2 |
|---|---|---|
| Question | Can AI perform legal tasks? | Can AI reliably judge AI-produced legal work? |
| Status | Published at NeurIPS 2023 | Ongoing project |
| Scope | 162 tasks across six types of legal reasoning, 20 models evaluated | Datasets of instructions, model responses and attorney preferences |
| People | 40 collaborators | Open to contributions; task-data contributors are credited as co-authors |
Example task categories
The project page lists these as illustrative, not exhaustive.
- Legal education: law school exam grading
- Depositions: question strategy evaluation
- Brief writing: argument quality
- Contract drafting: clause quality
- Regulatory analysis: compliance assessment
- Legal research memos
- Client advice letters
- Due diligence: contract review
- Demand letters
- Sentencing advocacy: mitigation memoranda
How to contribute
Task ideas. If you can identify a legal task where AI evaluation matters, describe the task and why it matters. No data is required. Not every submission can be included.
Task data. If you also have instructions to contribute that meet minimum quality standards, they can be turned into benchmarks. Contributors who provide task data are credited as co-authors on LB2.
Contributions go through the project’s contribution form.
Snorkel AI’s role
Snorkel AI supports LegalBench 2 through its Open Benchmarks Grants program, which backs open benchmarks, red-teaming, and fellowships for open AI evaluation. See every funded benchmark.
For teams building or buying legal AI, LB2 is a way to find out whether an AI grader can be trusted to stand in for attorney review. The project is still collecting tasks and data, and no results have been published yet.

