Reinforcement Learning With Verifiable Rewards: How Data and Verifiers Shape RLVR

Learn how RLVR uses verifiable rewards to train models, how training data and verifier design shape performance, and where RLVR still struggles with agents and long-horizon tasks.

---

TL;DR

• RLVR trains models using rewards from outcomes that can be checked programmatically, such as correct answers, executable code, or valid structured outputs.
• Verifiability alone does not produce effective training. Task composition, difficulty, verifier design, and the available optimization budget shape what the model learns.
• In Snorkel’s experiments, 100 mixed-difficulty examples reached 44.2% mean test accuracy, matching 500 easier examples. Increasing the mixed dataset to 500 examples under the same training budget reduced accuracy to 35.5%.
• Training reward should be evaluated separately from held-out performance because a model can improve against the verifier without improving on unseen tasks.

Reinforcement learning with verifiable rewards (RLVR) has emerged as a practical approach for training reasoning models on tasks with objectively checkable outcomes.

RLVR can provide consistent training feedback without requiring a human to evaluate every response. The approach has become especially relevant as researchers train models for more complex reasoning and long-horizon tasks.

Reinforcement learning with verifiable rewards (RLVR) promises a straightforward training loop: let a model attempt a task, check the result programmatically, and reinforce the responses that pass.

Reliable RLVR performance still depends on the surrounding training system. Data composition and verifier design influence what the model learns, which behaviors receive credit, and which failures remain undetected.

What Is RLVR?

Reinforcement learning with verifiable rewards (RLVR) is a post-training approach that rewards a model when its output satisfies an objectively checkable condition. A verifier evaluates the result and converts that judgment into a reward signal used to update the model.

RLVR works most cleanly on tasks with clear ground-truth answers. Mathematical problems can be checked against known solutions, code can be executed against tests, and structured outputs can be validated against schemas or formatting rules.

DeepSeek-R1 helped popularize this approach by using rule-based rewards for mathematical reasoning and code generation. Correct answers and valid outputs produced direct training signals without requiring a human to score every response.

How Does RLVR Work?

RLVR trains a model through repeated attempts, automated verification, and policy updates. A typical training loop follows these steps:

1. Select a training task. The model receives a prompt from a dataset with an outcome that can be checked reliably.
2. Generate multiple responses. The model attempts the task several times, producing different answers or reasoning paths.
3. Verify each outcome. A programmatic verifier checks the response against the task’s success criteria. This may include answer correctness, code execution, output structure, or other deterministic checks.
4. Assign a reward. Responses receive higher rewards when they satisfy the verifier. Some systems combine several reward components, such as correctness and valid formatting.
5. Update the model. The model’s policy is adjusted to make higher-reward behavior more likely in future responses.

This loop repeats across the training dataset. The model does not receive a written demonstration of the correct reasoning process for every task. It instead learns from the reward attached to the outcomes it produces.

RLVR vs. RLHF: What Is the Difference?

RLVR and reinforcement learning from human feedback (RLHF) both use rewards to reinforce model behavior. They differ in how those rewards are produced.

In a conventional RLHF pipeline, people compare model responses and indicate which ones they prefer. Those judgments are used to train a reward model that estimates how people may evaluate future responses. RLVR calculates rewards from predefined conditions that can be checked automatically.

	RLVR	RLHF
Feedback source	Programmatic checks, tests, rules, or environment state	Human demonstrations or preference judgments
Reward represents	Whether the output satisfies defined success criteria	Whether the output aligns with learned human preferences
Reward calculation	A verifier evaluates the model’s output directly	A reward model commonly predicts human preference
Best suited to	Tasks with outcomes that can be checked reliably	Tasks whose quality depends on judgment, interpretation, or preference
Primary failure risk	The verifier rewards an incomplete proxy for the intended behavior	The reward model poorly represents human preferences or annotation diversity

RLVR does not remove human judgment from training. People still select the task distribution, define acceptable outcomes, construct the verifier, and determine whether its rewards represent the intended behavior.

Verifiable feedback is most useful when success can be specified precisely. Qualities such as tone, helpfulness, and contextual appropriateness are harder to reduce to deterministic checks because several responses may be acceptable for different reasons.

The two approaches can also be combined. Verifiable rewards can train capabilities with measurable outcomes, while human feedback can shape behavior that depends on preference or interpretation. The resulting model still depends on the quality of both the training tasks and the signals used to evaluate them.

How Data and Verifiers Shape RLVR

In our recent study, Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes, we examined how the size and composition of RLVR training data affect learning when data and compute are limited.

To study these questions in controlled settings, we generated reasoning problems whose correct answers could be calculated programmatically. The tasks ranged from applying several numerical operations to a sequence of integers to reasoning over graphs and tracking objects through a two-dimensional environment.

One numerical problem, for example, might ask the model to consider the integers from 1 to 100, keep only the even values, then keep only those divisible by three, and report how many remain. The answer can be calculated exactly, allowing the verifier to score the model without human judgment.

We trained the Qwen3-4B model on datasets that varied in size and difficulty. Some contained only easier problems, while others combined easy, medium, and hard examples. This setup allowed us to examine whether adding more examples helped, whether the distribution of those examples changed the result, and when difficult examples stopped producing useful rewards.

Training Data Determines What the Model Practices

Every training prompt gives the model an opportunity to attempt a behavior and receive feedback. The distribution of those prompts determines which skills are reinforced repeatedly and which receive little or no training signal.

A dataset can contain many examples while still covering a narrow range of behavior. Prompts may differ in wording or surface details while requiring nearly identical reasoning, making the dataset appear more diverse than it is.

Useful diversity comes from varying the task structure, required reasoning, output constraints, and conditions under which a response should fail. These variations expose the model to different behaviors rather than repeated versions of the same problem.

Across these experiments, training sets with the same number of examples could produce different learning behavior depending on how those examples were distributed across difficulty levels. Dataset size alone did not determine how well the resulting model generalized.

Task Difficulty Changes the Learning Signal

In RLVR, a training example is useful only when the model can attempt it and the verifier can distinguish a better outcome from a worse one. Task difficulty affects both parts of that process.

When nearly every attempt succeeds, the model receives repeated confirmation for behavior it may already perform reliably. When nearly every attempt fails, the verifier returns little variation in reward and provides limited evidence about which behavior should be reinforced.

We defined difficulty empirically based on how often a panel of models solved each generated problem. We then compared training sets containing only easier examples with sets that mixed easy, medium, and hard examples.

• Easier tasks provide a stable starting signal. The model can complete enough attempts for successful behavior to receive credit. As performance improves, however, repeatedly training on similar easy examples can produce narrower gains because those examples introduce few new demands.
• More difficult tasks extend the behavior being trained. They require the model to apply its existing capabilities under less familiar or more demanding conditions. A dataset containing multiple levels of complexity can therefore reinforce reliable behavior while also creating opportunities to learn something new.
• Difficulty can exceed the model’s useful training range. When tasks regularly produce incomplete, invalid, or unverifiable responses, the reward signal becomes dominated by failure. The model receives a negative outcome, but little information about which part of its behavior should change.

Models trained only on easier problems could still improve on more difficult held-out examples, although the strength of that generalization varied across task families.

The useful difficulty range depends on the model and the training setup. Examples should introduce behavior the model has not mastered while remaining solvable often enough to generate meaningful rewards.

Data Composition Changed Sample Efficiency

The effect was clearest in the multi-step numerical problems introduced above. With 100 mixed-difficulty training examples, the model reached 44.2% mean test accuracy, matching the result achieved with 500 easier examples.

Mean test accuracy by training-set size and difficulty distribution. One hundred mixed-difficulty examples matched the accuracy achieved with 500 easier examples. Source: Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes.

The advantage did not continue as more mixed examples were added. Under the same fixed 300-step training budget, accuracy fell to 35.5% with 500 mixed-difficulty examples. Larger datasets received fewer optimization updates per example.

Verifier Design Determines What Counts as Success

A verifier is the part of an RLVR system that converts a model response into a training reward. It identifies the answer or output being evaluated, checks it against programmatic ground truth or another deterministic condition, and translates the result into a numerical score.

That score can represent more than final correctness. In our training setup, some task-specific rewards combined correctness with response-format bonuses and penalties for unnecessarily long reasoning. Another used binary exact-match correctness while allowing several valid response formats during answer extraction. These choices determined which aspects of each response contributed to its training reward.

A condition omitted from the verifier receives no explicit credit during training. A weak proxy can also reward a response that passes the check without demonstrating the intended behavior. Verifier design therefore requires deciding which aspects of success need separate checks and testing whether those checks remain reliable as the model improves.

What Effective RLVR Training Requires

RLVR performance does not depend on the reward function alone. The training data, task difficulty, verifier, and optimization budget determine whether the model receives useful feedback and whether improvements transfer beyond the training set.

Training decision	What to determine	Why it affects RLVR
Task distribution	Which behaviors, reasoning structures, and failure conditions appear in training	The model can only practice behavior represented in the task set
Difficulty	Which examples the current model can sometimes solve but has not mastered	Tasks that are consistently trivial or consistently unsuccessful provide limited learning signal
Verifier	Which parts of success receive credit and which valid outputs are accepted	The model optimizes the implemented check, including its omissions and shortcuts
Training budget	How dataset size, rollout length, and update count fit the available compute	Adding examples can reduce how often each example contributes to optimization

Evaluation should remain separate from the reward used during training. Held-out tasks, difficulty slices, and checks outside the training verifier are needed to determine whether reward improvements reflect broader capability rather than better optimization of the training setup.

Frequently Asked Questions About RLVR