MedPAIR is led by Yuexing Hao, a researcher at Microsoft, co-founder and COO of MatrAIx, and formerly a postdoc in MIT’s Healthy ML Group, with collaborators at MIT and other institutions. Snorkel AI supports the project through the Open Benchmarks Grants program.
What is MedPAIR?
LLMs can answer clinical exam questions well, but a correct answer does not show the model used the evidence a physician would. MedPAIR asks whether physicians and AI agree on what matters in medical question answering.
The dataset compares physician trainees and LLMs on clinical relevance. Physicians label every sentence in a patient case as highly relevant, low relevance or irrelevant to the question, and six LLMs label the same sentences. As a result, comparing the two shows where models rely on different evidence, and whether trimming a case to the sentences physicians call relevant changes a model’s answer.
~1,300
QA pairs annotated (933 in the filtering analysis)
59
physician annotators (52 trainees, 7 board-certified)
6
LLMs labeled the same sentences
50–60%
physician–LLM agreement on relevant sentences
How MedPAIR works
| Component | Description |
|---|---|
| Instructions | USMLE-style multiple-choice questions, each with a patient profile, a question and answer options, drawn from four public benchmarks (MMLU, JAMA, MedXpertQA and MedBullets). |
| Responses | Physicians label each sentence in the profile as highly relevant, low relevance or irrelevant. Only annotators who first answered the question correctly count. Six LLMs, open and closed, give self-reported relevance labels and context scores. |
| Scoring | Physician labels are aggregated by majority vote. The benchmark compares human and model relevance, then tests whether keeping only relevant sentences preserves answers, using a “spurious rate” that measures how often a correct answer flips. |
Key findings and status
MedPAIR is in development: the grant supports an expansion with more questions, multimodal and multilingual data, and ratings from board-certified physicians. However, no results from the expansion have been published. Findings from the initial dataset were presented at Snorkel’s Reading Group in October 2026 and in the project paper, an oral at the NeurIPS 2025 Workshop on Socially Responsible and Trustworthy Foundation Models:
- First, physicians and LLMs agree on roughly 50 to 60 percent of the sentences.
- Second, filtering out sentences that physician trainees labeled irrelevant improved accuracy for some models but not all. However, at least one MedGemma result got worse.
- Third, keeping only the highly relevant sentences flipped 46.3% of Qwen 72B’s previously correct answers to incorrect, against 14.9% for Llama 70B.
Physicians and LLMs agree on only about 50 to 60 percent of the sentences that matter in a clinical question.
Watch: Yuexing Hao presents MedPAIR
Yuexing Hao presented MedPAIR at Snorkel AI’s Reading Group in San Francisco. Watch the talk below, or read the full Reading Group recap.
How to get involved
- Use the released labels and dataset to study physician–model agreement.
- Snorkel AI recruited the annotators for the original dataset. In addition, clinicians interested in expert review work can join the Snorkel Expert Contributor community.
Snorkel AI’s role
Snorkel AI supports MedPAIR through Open Benchmarks Grants, a program that gives selected open-benchmark teams expert data, research and engineering support, and compute. See every funded benchmark.
Explore related benchmarks: JudgmentBench, which compares expert and model judgment, and KoraBench, a safety benchmark with expert review.

