Checking the Rubrics: RIFT Results on Five Public Benchmarks

Rubric-based evaluation is now a common way to score the outputs of language models and agents (Hashemi et al., 2024; Wei et al., 2025), and rubrics also serve as the source of reward signals in training (Gunjal et al., 2025; Liu et al., 2026; Ding, 2026). In both uses the rubric is the measurement instrument, and the validity of any resulting score depends on how well the rubric represents the construct the task is meant to assess (Messick, 1989). Rubric quality is therefore rarely examined directly. It is usually inferred from downstream evidence, such as the rate of agreement between automated judges and human raters (Hashemi et al., 2024; Shen et al., 2026) or the gains observed in models trained on rubric-derived rewards (Gunjal et al., 2025; Liu et al., 2026). Both measures confound the rubric with the model that applies it, and neither can detect a rubric that is internally consistent and interpretable yet misaligned with the task. A smaller body of recent work evaluates rubrics directly (Arabzadeh et al., 2024; Pan et al., 2026; Siro et al., 2026; Hong et al., 2026; Roy et al., 2026) and reports, for example, that machine-generated rubrics diverge from human-authored ones (Dhole & Agichtein, 2026).

RIFT (RubrIc Failure mode Taxonomy) addresses this gap by evaluating the rubric itself (Aich et al., 2026). It combines quantitative signals computed from a rubric’s content with a taxonomy of nine failure modes, so that a deviation in a signal can be tied to a named defect. This post applies RIFT to 147 tasks drawn from five public rubric-graded benchmarks, one each in finance, medicine, software engineering, professional advice, and reasoning. Results are reported by domain as the aim is to characterize rubric-writing practice in general and to help authors improve. The quality signals and the taxonomy make rubric failures measurable and give authors a shared vocabulary for discussing them.

In this sample, problems of coverage, alignment, and grounding were far more common than problems of atomicity and subjective wording. The review began with three expectations. The first was that atomicity and the anchoring of language would be largely sound. The second was that failures of coverage, alignment, and grounding would be more frequent. The third was that those failures would be most common in tasks where correctness depends on judgment. The first two held. Bundled requirements appeared in 7% of tasks and subjective wording in 10%, while missing criteria appeared in 50%, misaligned criteria in 48%, ungrounded verification in 37%, and redundancy in 35%. The third held for misalignment, which was highest in the judgment-heavy domains. Ungrounded verification showed the opposite pattern and was most frequent where answers are numeric or directly observable.

The RIFT framework

RIFT treats a rubric as a measurement instrument and evaluates it on two dimensions from measurement theory. Reliability concerns whether grading is reproducible. Content validity concerns whether the grading supports the intended interpretation of the score (Messick, 1989; Cook & Beckman, 2006).

The framework has two components. The first is a set of quality signals computed from the rubric’s text. A panel of simulated experts judges whether each criterion is essential to the task (Lawshe, 1975), and the criteria are then compared with the task’s topics and objectives to see how well they cover them. The point values are checked against the panel’s judgments to see whether the most important criteria carry the most weight (Raymond, 2001). The second is the taxonomy of nine failure modes, built with grounded-theory methods (Glaser & Strauss, 1967) from expert critiques of 85 rubrics. Table 1 previews the failure modes. Full definitions, decision rules, and examples for the signals and the taxonomy are in the paper (Aich et al., 2026).

Table 1. A preview of the nine RIFT failure modes and the quality dimension each belongs to.

These signals were validated with a controlled corruption experiment. Each failure mode was injected at known severity into 60 expert-authored rubrics, so the true defect was known for every variant. Every mode produced a distinguishable pattern across the signals. A multinomial logistic classifier over the signal changes identified the injected mode with 75.0% accuracy (chance 12.5%) under leave-one-task-out cross-validation. An LLM judge (Zheng et al., 2023) given the same rubrics reached 56.7%. The weight-calibration signals were validated in the same way.

The Five Domain Analysis

A total of 147 tasks were reviewed, with 27 to 30 tasks each from five public benchmarks. Each benchmark corresponded to one domain (seen in the table below).

For each task and failure mode, the analysis records whether the mode applied, along with its severity, evidence quoted from the prompt and rubric, and the RIFT signal values behind the call. The percentages below show the share of tasks where a mode was flagged. Tasks where a mode appeared only as a minor issue are not included in these percentages.

Two limits apply to the sampling and analysis. With about 30 tasks per domain, the 95% confidence interval on a rate near 50% spans roughly plus or minus 18 points, so between-domain comparisons are indicative and the pooled estimates across all 147 tasks are more reliable. Hackability is reported separately and discussed more below. Table 2 shows the prevalence of the remaining eight modes by domain.

Table 2. Share of reviewed tasks flagged for each failure mode, by domain. Columns run roughly from tasks with checkable answers (left) to judgment-heavy tasks (right), and the last column pools all 147 tasks.

Results

Rubrics are getting more reliable

Non-atomic criteria appeared in 11 of 147 tasks (7%) and subjective language in 14 of 147 (10%), making these the two least frequent modes. Atomicity supports reliable partial credit, because a single item such as “clear, accurate, comprehensive, and well-cited” cannot be scored consistently across graders (Brookhart & Chen, 2015; Brookhart, 2018). Bundling was absent from the professional advice sample and present in at most 10% of tasks in every other domain. This shows that rubric writers and rubrics are showing improvement at measuring a single criterion at a time and enhancing reliability.

Subjective language was found in 0% of software engineering tasks, 3% of professional advice tasks, 4% of finance tasks, and 7% of medicine tasks. In these domains, criteria were generally tied to an observable or checkable outcome. This suggests that rubrics for verifiable rewards or objective tasks are becoming more objective. The reasoning domain was the exception at 33%. Its tasks require the weighing of competing considerations, which makes anchoring harder, and one criterion asks whether the response “weighs the effect of following the therapist’s advice” without specifying which effects are relevant.

Frequent failure modes

Missing criteria (50% of tasks). This was the most frequent mode overall. It appeared in 90% of medical tasks (27 of 30) and in 27% to 48% of tasks in the other domains. In most cases the omission was direct. In a reasoning task, the prompt asked directly whether the user should take a particular action, but every criterion checked only that a relevant consideration was mentioned, so a response that covered every consideration without answering the question passed. Two of the three issues rated critical among the examples reviewed were missing criteria. The remaining criteria in such rubrics can each be sound, so the gap is visible only when the rubric is read against the prompt. Omitted requirements leave the construct under-represented (Messick, 1989). This matches the signature observed for the mode in the corruption experiment, where the surviving criteria stayed individually healthy and only the composition of the set changed.

Misaligned criteria (48%). A criterion is misaligned when it grades an objective the task did not specify or embeds an incorrect assumption. In one medical task, a criterion weighted at -5 penalized responses for attributing postoperative cognitive dysfunction to anesthesia, although this is a recognized complication and the patient’s own history raised it. In a software task that asked whether a reported bug was real, a criterion presupposed that it was, so a correct conclusion of misunderstanding would have lost credit. And in a third misaligned example, a finance criterion fixed the conclusion of a task that asked the responder to reach its own judgment. Criteria that grade an unrequested objective add construct-irrelevant variance (Haladyna & Downing, 2004). Rates were 11% in finance and 27% in software engineering, against 57% in professional advice, 70% in reasoning, and 73% in medicine. These criteria may not raise flags when read alone, but problems appear when the criterion is compared with the prompt or checked against domain knowledge, and the errors become clearly apparent to a reader with specialist knowledge.

Ungrounded verification (37%). These criteria require a fact to be checked without supplying a reference, tolerance, or procedure for checking it. Rates were 74% in finance and 57% in software engineering, against 20% in medicine and 3% in reasoning. Two forms occurred. In the first, a specific answer was demanded with nothing to verify it against, as in an exact price change with no reference closing prices. In the second, a criterion asserted a fact the prompt never established, as in a requirement for an echocardiogram in a patient with no cardiac symptoms, with no guideline cited. The corruption experiment found this mode to have the weakest signal footprint of the nine, so detection relies largely on reading each criterion against the task instruction.

Redundant criteria (35%). Redundancy was most frequent in reasoning (73%) and nearly absent in software engineering (3%). Showing that for open ended and judgement based tasks rubrics still continue to measure repetitive things. In one medical task, a criterion awarded +10 for stating that a medication could be continued before the procedure and another deducted 10 for suggesting it be stopped, which scores a single judgment twice from opposite directions, so one statement moved the score by 20 points. Duplicated scoring doubles the weight on one behavior without an explicit weighting decision, and pairs of this kind resemble thoroughness when criteria are read one at a time. Near-duplicate items are a recognized source of redundancy in instrument design (Boyle, 1991).

Mis-calibrated weighting (29%). Point values allowed a response to fail the core requirement and still score well. Rates ranged from 7% in software engineering to 57% in medicine. In one finance rubric totaling 18 points, the final conclusion, which was the task’s main deliverable, received 3 points, while nine criteria checking intermediate figures received 1 point each. A response with every intermediate step correct and the wrong conclusion still scored 83%. Text-based signals cannot detect this failure, since criterion wording can be sound while the points are misallocated. In the paper’s sample of 48 scoreable expert-authored rubrics, 10 had weights that ranked criteria against the panel’s view of necessity, which is the sense in which weights act as a claim about relative importance (Raymond, 2001).

Cross-domain patterns

Failure mode tracked how checkable the answer is. The finance and software engineering domains failed mostly on grounding, because their criteria call for specific values or behaviors and often omit the reference needed to verify them. Medicine, professional advice, and reasoning failed mostly on alignment. Medicine also showed high rates of missing criteria, which is consistent with prompts that contain many patient-specific constraints and answers that depend on context.

Conclusions

Across the five domains, the mechanical aspects of rubric construction were generally sound. Criteria were largely atomic, and wording was anchored wherever answers can be checked. The remaining failures concerned coverage, alignment, grounding, and duplicate scoring. These are validity problems, and correcting them requires familiarity with both the prompt and the subject matter.

Signals computed from the rubric can direct reviewers to the tasks and criteria most likely to be defective, and the taxonomy supplies a precise vocabulary for describing each defect. Domain experts remain necessary to decide whether a criterion is wrong and how it should be repaired. Snorkel’s Open Benchmarks Grants (OBG) program supports open benchmark development with expert data and research collaboration. Applying a RIFT-style review to a rubric before release, with specialists examining the flagged criteria, offers a low-cost check on what the rubric measures.

Notes

1. Hackability. Hackable was flagged in 83% of reviewed tasks (122 of 147), with rates of 63% in finance, 70% in software engineering, 83% in professional advice, 97% in medicine, and 100% in reasoning. Because it appears in nearly every benchmark, it separates domains poorly and is excluded from the main comparison. It remains important. It produced the largest effect in the paper’s corruption study (d = -2.83 on mean content validity ratio), and a rubric whose top score can be reached by asserting correct-sounding content without supporting reasoning is a substantive measurement risk. Excluding it leaves the ordering of domains by average prevalence unchanged (medicine, reasoning, professional advice, finance, and software engineering, at 36%, 33%, 32%, 25%, and 18% across the eight modes shown, and at 43%, 40%, 37%, 29%, and 24% with Hackable included).
2. Counting and sample. Percentages count reviewed tasks in which a mode was flagged. Finance has 27 reviewed tasks due to availability, and each other domain has 30 reviewed tasks.

References

• Aich, A., Qi, Z., Dickens, C., Pham, D., Sharma, E., Viktorov, J., Dsouza, A., Parchami, A., Sala, F., & Varma, P. (2026). The Hitchhikers Guide to Rubric Quality Understanding and Enrichment. arXiv:2604.01375.
• Arabzadeh, N., Huo, S., Mehta, N., Wu, Q., Wang, C., Awadallah, A. H., Clarke, C. L. A., & Kiseleva, J. (2024). Assessing and verifying task utility in LLM-powered applications. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 21868-21888.
• Boyle, G. J. (1991). Does item homogeneity indicate internal consistency or item redundancy in psychometric scales? Personality and Individual Differences, 12(3), 291-294.
• Brookhart, S. M. (2018). Appropriate criteria: Key to effective rubrics. Frontiers in Education, 3, 22.
• Brookhart, S. M., & Chen, F. (2015). The quality and effectiveness of descriptive rubrics. Educational Review, 67(3), 343-368.
• Cook, D. A., & Beckman, T. J. (2006). Current concepts in validity and reliability for psychometric instruments: Theory and application. The American Journal of Medicine, 119(2), 166.e7-166.e16.
• Dhole, K. D., & Agichtein, E. (2026). RubricRAG: Towards interpretable and reliable LLM evaluation via domain knowledge retrieval for rubric generation. arXiv:2603.20882.
• Ding, L. (2026). AdaRubric: Task-adaptive rubrics for reliable LLM agent evaluation and reward learning. arXiv:2603.21362.
• Glaser, B. G., & Strauss, A. L. (1967). The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine.
• Gunjal, A., Wang, A., Lau, E., Nath, V., He, Y., Liu, B., & Hendryx, S. (2025). Rubrics as rewards: Reinforcement learning beyond verifiable domains.
• Haladyna, T. M., & Downing, S. M. (2004). Construct-irrelevant variance in high-stakes testing. Educational Measurement: Issues and Practice, 23(1), 17-27.
• Hashemi, H., Eisner, J., Rosset, C., Van Durme, B., & Kedzie, C. (2024). LLM-Rubric: A multidimensional, calibrated approach to automated evaluation of natural language texts. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 13806-13834.
• Hong, H., Li, Y., Chen, J., Huy, L. G., Ananiadou, S., Kim, J., & Lin, C. (2026). Can LLMs write reliable rubrics? A meta-evaluation for experiment reproduction. arXiv:2607.12835.
• Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563-575.
• Liu, T., Xu, R., Yu, T., Hong, I., Yang, C., Zhao, T., & Wang, H. (2026). OpenRubrics: Towards scalable synthetic rubric generation for reward modeling and LLM alignment. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 17417-17437.
• Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational Measurement (3rd ed., pp. 13-103). American Council on Education and Macmillan.
• Pan, T., Lin, X., Yang, W., He, Q., Chen, S., Qi, L., Xu, W., Feng, H., Xu, B., & Xiao, Y. (2026). RubricEval: A rubric-level meta-evaluation benchmark for LLM judges in instruction following. arXiv:2603.25133.
• Raymond, M. R. (2001). Job analysis and the specification of content for licensure and certification examinations. Applied Measurement in Education, 14(4), 369-415.
• Roy, S., Pujari, R., Kumarage, T., et al. (2026). PReMISE: Policy rubrics as measurement specifications for LLM judges. arXiv:2605.30803.
• Shen, W. F., Qiu, X., Whitehouse, C., et al. (2026). Rethinking rubric generation for improving LLM judge and reward modeling for open-ended tasks. arXiv:2602.05125.
• Siro, C., Aliannejadi, P., & Aliannejadi, M. (2026). Learning to judge: LLMs designing and applying evaluation rubrics. Findings of the Association for Computational Linguistics: EACL 2026, 6371-6389.
• Wei, Y., Pearl, D., Beckman, M., & Passonneau, R. J. (2025). Concept-based rubrics improve LLM formative assessment and data synthesis. arXiv:2504.03877.
• Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track.