When Partial Credit Opposes Completion

AI agents are increasingly trusted with long, open-ended work. They fix bugs across large repositories, operate terminals for hours, and chain numerous tool calls toward a single goal. The strongest models now solve a meaningful share of such tasks, yet on the hardest ones failure is still the common outcome, and closing that gap is one of the central problems in post-training.

The standard recipe is reinforcement learning against a verifier. The agent attempts a task many times, an automatic checker scores each attempt, and the model is pushed toward the attempts that scored best. When success is rare, this recipe runs into a basic problem. If the checker reports only whether the task was fully solved, almost every attempt on a hard task scores zero, and a group of identical zeros teaches nothing. The natural remedy is partial credit, scoring each attempt by the fraction of requirements it met, and it is a common choice in practice.

Used naively, partial credit can work against the very goal it is meant to serve. Even though it is built from the same checks that define success, a model trained on it can learn to satisfy the easy checks while giving up on the route that satisfies them all. Blending partial credit with the pass/fail signal does not solve this, since a failed attempt can still be reinforced right alongside a successful one unless almost all of the weight sits on pass/fail. We introduce binary-gated GRPO, which removes this conflict while keeping what partial credit has to offer. Whether partial credit then helps or hurts completion comes down to a single quantity that the scores alone cannot reveal. The good news is that with binary-gated GRPO, harm requires a specific combination of conditions, and in a natural setting partial credit can only help.

The verifier rarely grades a single final answer. More often, it checks many conditions once the attempt is over. A coding agent may face a test suite. A tool-use agent may face a checklist. A rubric may be decomposed into atomic, pre-specified requirements. In each case, we assume deployment cares only whether the agent satisfied every required condition.

That all-pass objective is the right one. It is also an aggressively lossy summary of what the verifier observed.

Suppose a verifier evaluates $M$ binary checks. Let $Z_m\in \{0,1\}$ record whether check $m$ passes, let positive weights $w_m$ sum to one, and define

$$R=\sum _{m=1}^M{w}_m{Z}_m,B=\prod _{m=1}^M{Z}_m.$$

Here $B$ is the binary all-pass return and $R$ is the terminal fractional return, which makes the partial credit described above precise. Because every weight is positive, $B=\mathbf{𝟏}\{R=1\}$. The fractional return therefore retains the all-pass outcome and can preserve variation among failures.

This is why fractional return is compelling. It is derived directly from the atomic conditions whose conjunction defines success, although the weights still choose how to compare non-all-pass outcomes. The full check vector says which requirements a rollout met, while the scalar $R$ records only the weighted fraction.

$B$ and $R$ are returns, not rewards. In reinforcement learning, a reward is defined for a single transition, one action taken in one state. The verifier instead scores the finished trajectory as a whole, and a score of the whole trajectory is, by definition, a return. Calling such scores rewards is common, but it is a textbook mistake. We therefore refer to both as returns throughout.

We study the tension raised at the start for GRPO (Shao et al., 2024), a standard method for this kind of training. The question is not whether fractional return carries extra information, since it does, but whether GRPO can use it without trading completion for a more attractive form of failure.

GRPO samples a group of rollouts for each prompt and labels each rollout by how its return compares with the rest of its group. Two different things can go wrong. The first happens inside a single group. Fractional return can order trajectories differently from completion, and after centering, that disagreement can give a failed rollout a positive label, reinforcing it even when a successful rollout sits in the same group. The second is subtler and happens across groups. The same prompt yields a mixed group, containing both successes and failures, on one draw and an all-fail group on the next, and a rule that switches to fractional return only on all-fail groups changes what training optimizes, even when every single group looks fine.

In this post, we first make both sides of that tension precise. We then show that mixing fractional return with binary return does not remove the within-group conflict, and introduce binary-gated GRPO, which does. Next, we derive exactly what the cross-group problem does to training, identify the single quantity that decides its direction, and show that it is harmful only under specific conditions. We close with a practical rule for real training runs. Proofs of every proposition are collected in the appendix.

We work with fractional return throughout because it lets every claim be stated exactly. The central results do not depend on its particular form, however. Binary gating, what the all-fail fallback optimizes, and the quantity that decides whether it helps all apply unchanged to any score assigned to a whole trajectory, such as a sum of per-turn scores.

Why fractional return is tempting

Binary completion collapses every failure to the same label. A trajectory that passes nine of ten checks and a trajectory that makes no progress both receive $B=0$. That is appropriate as the deployment objective, but it discards information that may be predictive of how to reach completion.

Let $X$ denote one policy decision, comprising the full context available immediately before a sampled agent token together with that token. All expectations in this section are under one joint distribution of the decision $X$ and the trajectory that contains it. Define the centered completion-value target at that decision,

$${\mathrm{\Delta }}_B^{\pi }(X)=\mathbb{𝔼}[B\mid X]-\mathbb{𝔼}[B],$$

and the part of terminal fractional return that remains after pass/fail is known,

$$U_R=R-\mathbb{𝔼}[R\mid B]=\mathbf{𝟏}\{B=0\}(R-{\mu }_0),{\mu }_0=\mathbb{𝔼}[R\mid B=0].$$

For this nondegenerate discussion, assume both success and failure have positive probability. On successful trajectories $U_R=0$. On failures, it records whether a rollout achieved more or less fractional return than the average failure. These definitions let us state the information claim precisely.

Proposition 1 is an information result, not a guarantee for a particular policy-gradient estimator. It says that fractional return can carry information about the completion value of intermediate decisions that pass/fail labels alone cannot represent, and it identifies when that information improves prediction of that value. It does not say that pushing the policy toward higher fractional return helps completion. Carrying information and pointing an update in the right direction are different properties, and the rest of this post is about the gap between them.

Now make the GRPO setup precise. For one prompt, sample a group of $G\ge 2$ independent rollouts. For a scalar return $Y_i$ of rollout $i$, the centered label is

$$A_i=Y_i-\bar{Y},\bar{Y}=\frac{1}{G}\sum _{j=1}^G{Y}_j.$$

Throughout, GRPO means this raw centered label applied to the log-probability summed over each trajectory. A leave-one-out baseline gives the same label up to a factor of $G/(G-1)$. The original GRPO also divides each group by its standard deviation and averages token losses within each sequence. A later section says which results survive those choices. We call the rule $A_i=B_i-\bar{B}$ binary GRPO.

With binary all-pass return, every all-fail group has $B_i=\bar{B}=0$. Its return-derived task gradient is therefore exactly zero. On hard tasks, these groups can be common rather than exceptional, and a difficult prompt contributes no task gradient of its own until a rollout happens to complete.

Something has to fill that silence. Larger groups, curricula, demonstrations, and learned process reward models can all help, but each costs more or brings its own assumptions. Fractional return is the cheapest candidate, because the verifier already computes it.

The simplest options are to train directly on $R$, or to add $R$ beside $B$. Both are tempting, and both deserve a closer look.

Fractional return can silently prefer failure

The attraction of fractional return makes the following failure easy to miss. The score is built from the conditions that define success, yet it can still induce a direct preference for a trajectory that never completes the task.

Consider a terminal task with five equally weighted checks. Under policy ${\pi }_A$, half the rollouts pass three checks and half pass two. Thus

$${\mathbb{𝔼}}_{{\pi }_A}[R]=0.5,{\mathbb{𝔼}}_{{\pi }_A}[B]=0.$$

Under policy ${\pi }_B$, an implementation that preserves every required behavior completes all five checks with probability $0.2$, while the other rollouts pass none. Thus

$${\mathbb{𝔼}}_{{\pi }_B}[R]=0.2,{\mathbb{𝔼}}_{{\pi }_B}[B]=0.2.$$

Fractional return prefers ${\pi }_A$, the policy that never finishes the task. Binary completion prefers ${\pi }_B$. Group centering does not change this. For independent rollouts, the expected GRPO update on $R$ is exactly $(G-1)/G$ times the gradient of $\mathbb{𝔼}[R]$ (Appendix A.4), so GRPO trained on $R$ inherits this preference.

This is not a pathology caused by using an imprecise evaluator. Both returns came from the same checks. The problem is that satisfying many checks can be easier than preserving the route that satisfies all of them. For example, an agent can implement visible new behavior while breaking a legacy invocation that is required for full completion.

The literature would call this reward hacking, even though the quantity being gamed here is a return. The example supports a sharper claim, and a more useful one. Even though fractional return is computed from the very same checks that define success, it can still rank a shortcut above completion.

Mixing still reinforces failure

One might be tempted to retain binary completion and naively mix it with fractional return,

$$A_i^{\mathrm{m}\mathrm{i}\mathrm{x}}=\beta (B_i-\bar{B})+(1-\beta )(R_i-\bar{R}),0\le \beta <1.$$

While this appears conservative when $\beta$ is large, it turns out to be far from conservative in the rare-success regime.

Consider $G=16$ rollouts. One completes, one failure obtains $R=0.8$, and the remaining fourteen failures obtain $R=0.5$. The group means are $\bar{B}=1/16$ and $\bar{R}=0.55$, so with $\beta =0.7$ the $R=0.8$ failure receives

$$0.7(0-\frac{1}{16})+0.3(0.8-0.55)=0.03125.$$

It receives a positive label despite a successful rollout in the same group. In the raw score-function term, a positive label pushes the failed rollout’s sampled actions toward higher probability. The group has already demonstrated completion, yet the mixture still reinforces a failure direction. We call a positive label on a failure in a mixed group a sign reversal.

This is not a peculiarity of the seventy-percent example. A sign reversal can occur in any mixed group, and the next result gives the exact condition.

In the example above, $R_i-{\bar{R}}_{-i}=0.8-8/15=4/15$, so the failure is reinforced for every $\beta <0.8$. Other verifiers can do worse. Every success scores $R=1$, which pulls ${\bar{R}}_{-i}$ up, but a failure can come arbitrarily close to $R=1$ while its other failed peers score zero. As a result, no choice

$$\beta <1-\frac{s}{G-1}$$

is safe for every verifier. With $G=16$ and one success, the binary coefficient must be at least $14/15\approx 93.3\%$ to rule out every such reversal. A specific verifier sets a slightly lower bar. If its failures can score anywhere from 0 up to $r_{\max }<1$, the threshold is $1-s/((G-1)r_{\max })$. With five equal checks, $r_{\max }=0.8$, and for the same group size and one success the threshold is $11/12\approx 91.7\%$. The criterion cannot fire at all when a group has a single failure, because that failure sits below peers that all have $R=1$.

A binary weight close enough to one does prevent the reversal. The trouble is where that weight is needed. When a group contains many successes, a modest weight suffices. When successes are rare, which is exactly the regime where fractional return was supposed to help, most mixed groups contain a single success, and the weight needed to rule out every reversal is $1-1/(G-1)$, already $6/7$ at $G=8$. A weight that large leaves fractional return only a minor role. Mixing is safe exactly where it is unnecessary, and unsafe exactly where it is tempting.

Let completion decide first

Every fixed mixture sets an exchange rate between fractional return and completion, while the deployment criterion gives completion strict priority. An exchange rate is the wrong kind of object. We propose a composition that expresses the priority directly, which we call binary-gated GRPO. It lets completion decide wherever it can, and admits fractional return only where completion is silent.

Let $s={\sum }_i{B}_i$ be the number of successful rollouts in the group. The binary-gated terminal label is

$$A_i^{\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{e}}=\{\begin{array}{cc}{B}_i-\bar{B},\hfill & 0<s<G,\hfill \\ {\eta }_0(R_i-\bar{R}),\hfill & s=0,\hfill \\ 0,\hfill & s=G,\hfill \end{array}{\eta }_0\ge 0.$$

In a mixed group, completion already ranks every success above every failure. It does not rank failures against each other, and the binary-gated rule deliberately gives up that information, so that fractional return can never turn a failure into a positive example. In an all-fail group, binary completion ties every rollout, so fractional return is admitted as a tie-breaker. We call this all-fail branch the fallback. In an all-pass group, every rollout has $R=1$, so both returns are constant and the group is silent.

This removes the reversal that mixing allowed. A high-scoring failure can no longer receive a positive label in a group that already contains a completion.

What it does not control is what happens across groups. A high-scoring failure in an all-fail group still receives a positive label, while successes in other groups receive theirs, and no single group contains both. The next section shows that, on average, the steps these groups produce can lower completion, and a later section shows when that actually happens.

The all-fail switch changes what the update estimates

Each training step on a prompt draws one group from the current model, and whether that group is mixed or all-fail is a matter of chance. On a mixed draw, binary-gated GRPO steps along completion. On an all-fail draw, it steps along fractional return instead. Each step is sensible on its own, but every step starts from the model the previous one produced, so their effects on completion add up, and an all-fail step can undo part of what a mixed step gained.

It is tempting to call the all-fail branch “centered fractional return” and to treat the group mean as a harmless baseline. Applied to every group, centering is indeed a baseline, since in expectation it only rescales the gradient of the expected return. The all-fail branch is not applied to every group. It is applied only after every rollout has failed, and that selection changes what the fallback optimizes.

Write

$$p=\mathbb{𝔼}[B],q=1-p,{\mu }_0=\mathbb{𝔼}[R\mid B=0],c_G=\frac{G-1}{G}.$$

Here $p$ is the probability that a single rollout completes. For independent rollouts, $q^G$, the $G$th power of $q$, is the probability that all $G$ rollouts in a group fail.

Let $g_i$ be the trajectory score function for rollout $i$, namely the sum of log-policy gradients over its policy-controlled actions, and write $g$ for the score of a single generic rollout. Every gradient $\nabla$ is taken with respect to the policy parameters. Everything here refers to a single fixed prompt. Assume independent on-policy rollouts, $0<p<1$, and a verifier with no explicit parameter dependence. The results below concern the expected raw return-derived score estimator for one step, before clipping, group-standard-deviation normalization, or other optimizer transformations.

The expectation below is over the group drawn at a single step from the current model. It is the average step, with mixed and all-fail draws weighted by how often they occur, and its projection onto $\nabla p$ says whether a step taken from the current model raises completion on average, to first order in the step size.

The first term comes from mixed groups and is the intended completion gradient. The second comes from all-fail groups, which occur with probability $q^G$, and follows the mean fractional return among failures. It is not the gradient of all-pass completion, and it need not point in the same direction. When the two conflict, no single step reveals it, because each step contributes to only one of the two terms.

Proposition 4 also clarifies what binary-gated GRPO buys in expectation. No single group trades completion for fractional return, but averaged over draws, the two directions still combine, with weight ${\eta }_0{q}^G$ on $\nabla {\mu }_0$ relative to $\nabla p$. In expectation, this acts as an effective exchange rate, although it is never applied inside a group. The naive mixture, by contrast, applies a fixed exchange rate inside every group. It is ordinary centering of $\beta B+(1-\beta )R$, so its expected update is (Appendix A.4)

$$\mathbb{𝔼}[{\hat{g}}_{\mathrm{m}\mathrm{i}\mathrm{x}}]=c_G\left[\right(1-(1-\beta ){\mu }_0)\nabla p+(1-\beta )q\nabla {\mu }_0].$$

Two differences favor the binary-gated rule. Its weight on $\nabla {\mu }_0$ decays geometrically as successes become common, like $q^G$ rather than $q$, and it pays no $-(1-\beta ){\mu }_0$ penalty on the completion term. But when successes are rare enough that $pG\ll 1$, both $q^G$ and $q$ are close to one, and the weight each rule puts on $\nabla {\mu }_0$ is set mainly by its coefficient, ${\eta }_0$ or $1-\beta$. What binary-gated GRPO still guarantees in that regime is a property of every sampled group, namely that no failure is reinforced next to a success. In expectation, the direction $\nabla {\mu }_0$ is still there, and its sign is what matters.

Here is a minimal example. Let a one-dimensional policy parameter $\theta \in (0,1)$ and a constant $0<\epsilon \le 1$ produce three terminal outcomes.

One way to read it is as a sequence of choices. The policy first attempts the route that keeps completion possible and completes with probability $\epsilon \theta$. Otherwise it commits to the zero-return failure with probability $\theta$ or to the half-return failure with probability $1-\theta$. Then

$$p=\epsilon \theta ,{\mu }_0=\frac{1-\theta }{2}.$$

Increasing $\theta$ increases completion but decreases fractional return among failures. Since $dp/d\theta =\epsilon$, $d{\mu }_0/d\theta =-1/2$ and $q=1-\epsilon \theta$, Proposition 4 gives the expected binary-gated update

$$c_G[\epsilon -\frac{{\eta }_0}{2}(1-\epsilon \theta {)}^G].$$

For every positive ${\eta }_0$, a sufficiently small $\epsilon$ makes this a descent direction for completion. The fallback is not malicious. It is faithfully increasing conditional fractional return, which happens to prefer retaining the half-return failure mode.

When fractional return is safe, and when it actually helps

The previous section does not say that fractional return is usually harmful. It says that its direction must be checked.

Assume $\nabla p\ne 0$, and let

$$v_1=\mathbb{𝔼}[g\mid B=1],{\tau }_c=\frac{⟨\mathbb{𝔼}[(R-{\mu }_0)g\mid B=0],v_1⟩}{{\Vert v_1\Vert }^2}.$$

The vector $v_1$ is the mean score of successful rollouts, so it points along what the policy does when it completes. The vector $\mathbb{𝔼}[(R-{\mu }_0)g\mid B=0]$ points along what distinguishes high-return failures from the average failure, and it equals $\nabla {\mu }_0$ (Appendix A.4). So ${\tau }_c$ asks whether moving the policy toward better-scoring failures also moves it toward completion. If every failure receives the same fractional return, ${\tau }_c=0$. Unlike any statistic of the returns, ${\tau }_c$ is a statement about policy-gradient geometry, and we will see that identical returns can come with either sign.

These are different statements. “Still ascends” is not the same as “helps more than binary.” In the three-outcome example, the parameter is one-dimensional and ${\tau }_c=p(d{\mu }_0/d\theta )/(dp/d\theta )=-\theta /2$. The fallback therefore always pulls against completion there, and for small $\epsilon$ it overwhelms the completion term.

Proposition 5 also shows why ${\tau }_c$ has to be measured rather than guessed. Consider a policy that chooses among three strategies through a softmax over three logits, with probabilities 0.02, 0.49 and 0.49. The first strategy completes with probability 0.9 and otherwise ends in a near miss that scores 0.983. The second never completes and always ends in that near miss, and the third never completes and scores 0. This gives $p=0.018$ and ${\mu }_0=0.4925$. Now write the $j$th logit as ${\kappa }_j{\varphi }_j$ for a fixed positive constant ${\kappa }_j$, and take gradients with respect to $\varphi$. Choosing each ${\varphi }_j$ so that ${\kappa }_j{\varphi }_j$ equals the original logit leaves every probability, and hence every statistic of the returns, unchanged, but the gradients change. The table reports ${\tau }_c$ and the completion progress $1+{\eta }_0{q}^G{\tau }_c/p$ of Proposition 5 for $G=8$ and ${\eta }_0=0.5$, and Appendix A.7 gives the formulas behind it.

Nothing a practitioner can see in the returns distinguishes these rows, yet the fallback ranges from nearly eight times binary GRPO’s progress to pushing completion down. This dependence on parameterization is not a flaw in the criterion. Gradient updates are applied in whatever coordinates the optimizer uses, so their direction genuinely changes with the parameterization. It does mean that ${\tau }_c$ has to be measured in the model’s own coordinates rather than predicted from a toy model.

The factor $q^G/p$ is how often the fallback fires, divided by how much completion signal there is to compete with. Rare success amplifies the effect, and the sign of ${\tau }_c$ sets its direction. The same alignment enters the fixed mixture, whose normalized completion progress is $1-(1-\beta ){\mu }_0+(1-\beta )q{\tau }_c/p$, but because of the $-(1-\beta ){\mu }_0$ term it beats binary GRPO only when ${\tau }_c>p{\mu }_0/q$, a stricter condition than ${\tau }_c>0$.

This gives a concrete meaning to using fractional return correctly. It does not mean simply adding it after all failures. It means operating the fallback where ${\tau }_c$ is nonnegative, and turning it down or off where it is not.

Thus, in this regime, the terminal fractional fallback never reduces the expected completion-direction progress relative to binary GRPO, and whenever ${\eta }_0>0$ and $\nabla p\ne 0$ it strictly adds to it.

This is the positive case worth remembering. If fractional return reflects independent pieces of competence, higher fractional return among failures and more completion are compatible goals. The model is illustrative, and it makes no claim about arbitrary neural-network parameterizations. Within it, difficulty amplifies the fallback’s effect through $q^G/p$, but never sets its sign.

What can make ${\tau }_c$ negative is structure beyond difficulty. Competing strategies are one source. A shortcut can satisfy many visible checks while making a remaining requirement unreachable, as in the three-outcome example. Unequal weights are another, even when checks are independent. Improving a low-weight check raises $p$, but it also removes high-scoring near-misses from the failure set while adding little fractional return, and that can lower ${\mu }_0$. With weights $0.9$ and $0.1$ and pass rates $0.95$ and $0.5$, improving the second check does exactly that.

How serious is the cross-group problem?

The two problems differ in kind. The sign reversal of mixing is visible in individual groups. In the rare-success regime, most mixed groups contain a single success, and a high-scoring failure in such a group can be reinforced unless the binary weight is close to one. In the sixteen-rollout example, a 70% weight is not enough. The cross-group problem, by contrast, exists only in expectation, and it needs several conditions at once to do serious damage.

First, success must be rare. The fallback fires only on all-fail groups, which occur with probability $q^G$. At $p=0.5$ and $G=8$, that is one group in 256, and the rule is nearly inert.

Second, the alignment must be adverse enough. By Proposition 5, the update lowers completion only when ${\tau }_c<-p/({\eta }_0{q}^G)$, not merely when ${\tau }_c<0$. Difficulty alone never makes ${\tau }_c$ negative, which Proposition 6 proves for independent, equally weighted checks. It takes additional structure, such as competing strategies or unequal weights.

Third, the prompt must learn mostly from its own groups. Parameters are shared, so a prompt’s completion can rise through updates driven by other prompts. Suppose $K$ prompts share the parameters, write $p_k$, $q_k$ and ${\mu }_{0,k}$ for the quantities of prompt $k$, and let the update average the prompts’ binary-gated updates. By Proposition 4, the expected update is

$$\mathbb{𝔼}[\hat{g}]=\frac{c_G}{K}\sum _{k=1}^K(\nabla p_k+{\eta }_0{q}_k^G\nabla {\mu }_{0,k}),$$

and prompt $t$ gains completion at first order exactly when $⟨\nabla p_t,\mathbb{𝔼}[\hat{g}]⟩>0$. Prompt $t$’s own fallback term, which carries the whole single-prompt failure mode, is one of $K$ terms in this sum. The other prompts contribute $⟨\nabla p_t,\nabla p_k⟩$, their completion pressure on prompt $t$, and ${\eta }_0{q}_k^G⟨\nabla p_t,\nabla {\mu }_{0,k}⟩$, their fallback pressure on it. If the other prompts are unrelated to prompt $t$, so that both inner products vanish, the criterion is exactly the single-prompt one, however large $K$ is. If they are related, their terms add up across the other $K-1$ prompts while the own term remains a single term, so in a large corpus it is the relations between prompts, not prompt $t$’s own geometry, that decide whether the fallback helps it.

Fourth, the damage accumulates with repetition. The direction of each step does not depend on how often a prompt is visited, but the distance traveled along $\nabla {\mu }_0$ grows with the number of visits. A prompt visited $V$ times sees $VG$ rollouts in total, so a prompt whose completion rate is well below $1/(VG)$ will most likely never complete during the run. Its entire contribution is then the fallback term, with no completion term to anchor it.

Put together, the unfavorable corner is a small set of loosely related, rarely solved prompts whose fallback pulls against completion, trained for many epochs, while a large corpus of related tasks traversed a few times sits at the opposite corner. Binary-gated GRPO does not remove the cross-group problem, but the damage it can do is concentrated in the unfavorable corner.

None of this rules the problem out. The protection from related prompts depends on their fallback pressure on prompt $t$, and when those prompts are hard it is fragile. Their completion gradients are small, since $\nabla p_k=p_k{v}_{1,k}$, where $v_{1,k}$ is the mean score of prompt $k$’s successful rollouts, while their $\nabla {\mu }_{0,k}$ need not be. A slight adverse alignment between $\nabla {\mu }_{0,k}$ and $\nabla p_t$ can then outweigh all of the completion transfer, and these cross-prompt alignments have not yet been measured on real training runs.

What this analysis covers

Propositions 4 to 6 describe the expected update of the raw centered estimator for a single on-policy step. Common implementation choices change that in different ways.

• Group standard-deviation normalization. Dividing a group’s labels by a single positive number preserves every within-group sign, so Proposition 3 and the sign-reversal criterion for a mixed label survive. The expected-update results do not. Each all-fail group is divided by its own standard deviation, which varies with the group’s composition, so the all-fail branch no longer follows $\nabla {\mu }_0$, and the sign of its projection onto $\nabla p$ can differ from the raw estimator’s.
• Per-sequence length normalization. Averaging token losses within each sequence replaces the trajectory score by a length-weighted score, so the centering identity behind Proposition 4 no longer holds exactly. Varying lengths change the size of the update, and when length correlates with outcome, as it often does for agents, they can also change its direction.
• Preconditioned optimizers. An update $P\hat{g}$ with a fixed positive-definite preconditioner $P$ changes the relevant alignment to $⟨\nabla {\mu }_0,P{v}_1⟩/⟨v_1,P{v}_1⟩$, and its sign can differ from the Euclidean ${\tau }_c$. Adam’s preconditioner also depends on the gradients themselves, so this is a first approximation there.
• Dynamic sampling. Filtering out groups in which every rollout passes or every rollout fails, as in DAPO (Yu et al., 2025), keeps the binary update parallel to $\nabla p$ but gives no signal on a prompt that never succeeds. The fallback exists precisely to use the groups that dynamic sampling discards.
• Clipping and several epochs per batch. At the behavior policy the clipped surrogate has the same gradient as the unclipped objective, so the results describe the first step. They are not guarantees about later steps or about the final policy.

Propositions 1 to 5 hold for any score assigned to a whole trajectory. The verifier thresholds after Proposition 2, the comparison with the fixed mixture, Proposition 6 and the worked examples also use the structure of fractional return, namely that every success scores one and that the return is built from individual checks.

A practical rule

For a real training run, the first questions are operational rather than philosophical.

1. Is the fallback active? Measure the all-fail-group frequency and the fraction of all-fail groups with nonconstant $R$. If fractional return is constant within nearly every all-fail group, there is no fractional-return signal to use.
2. Which way does it pull? Estimate ${\tau }_c$ from rollouts the trainer already has. It needs the mean trajectory score over successes and the mean score over failures weighted by $R-{\bar{R}}_{\mathrm{f}\mathrm{a}\mathrm{i}\mathrm{l}}$, where ${\bar{R}}_{\mathrm{f}\mathrm{a}\mathrm{i}\mathrm{l}}$ is the mean fractional return of the failures. Each is the gradient of a weighted sum of log-probabilities, so each costs one backward pass. Noise inflates ${\Vert v_1\Vert }^2$, so compute the denominator from two disjoint halves of the successes. Under a preconditioned optimizer, take the inner products in the optimizer’s geometry. The distribution of $R$ alone cannot answer this question.
3. Know where it cannot be measured. ${\tau }_c$ requires successes, so it cannot be estimated on prompts that never complete, which are exactly the prompts where the fallback fires most. Count how many prompts never complete during the run, since for them the fallback is the only signal. Pooling across prompts that do complete estimates a corpus-level quantity, not the missing per-prompt one.
4. Does the model improve the right endpoint? Report all-pass success as the primary metric, and track $\mathbb{𝔼}[R\mid B=0]$ only as a diagnostic. Rising fractional return with declining completion is evidence of the failure mode above. Rising fractional return with flat completion is not, by itself, evidence of harm. On hard prompts that is also what healthy learning looks like at first. With ten independent checks each improving from 0.5 to 0.6, ${\mu }_0$ rises from 0.50 to 0.60 while $p$ moves only from 0.001 to 0.006.

The point is not to retreat to binary-only learning. On hard verifier tasks, that discards variation the verifier already provides. The point is to preserve the information in a decomposable verifier while refusing to let a fractional-return shortcut outrank completion.

Closing thought

Fractional return is valuable precisely because it is not arbitrary. It comes from the same checks that define success. In a benign competence regime, it gives completion-aligned signal exactly where binary GRPO is silent. But decomposition is not the same as alignment. Once an agent can reach the same checks through competing strategies, a terminal fractional return can optimize a more attractive failure instead of completion.

The right response is neither to discard fractional return nor to mix it blindly with a large binary coefficient. Letting completion decide every comparison it can removes the conflict inside each group. What remains lives across groups, and it matters mostly when a few hard, loosely related tasks are revisited many times. There, it is worth measuring ${\tau }_c$ to check that the fractional branch still points toward the endpoint that matters.

Binary gating is also a building block rather than a complete recipe. Behaviors that must never be reinforced, such as breaking tests that used to pass, can be made part of the completion criterion or can zero the fallback’s return, and Propositions 1 to 5 hold unchanged for the redefined returns. Nor does gating have to discard fractional return inside mixed groups. A binary-anchored label keeps every binary sign and lets fractional return rescale magnitudes within each outcome class (Appendix A.8). It carries the same guarantee as Proposition 3, but its mixed-group term is then no longer the pure completion gradient, so its expected update deserves the same scrutiny as the fallback.

References

• Shao, Z., et al. (2024). DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.
• Yu, Q., et al. (2025). DAPO: An open-source LLM reinforcement learning system at scale.

Appendix A. Proofs and derivations

This appendix collects the proofs of Propositions 1 to 6, the computation behind the three-strategy table, and the guarantee for the binary-anchored label.

A.1 Proof of Proposition 1

By construction,

$$\mathbb{𝔼}[U_R\mid B]=0.$$

Hence, for every $f(B)$,

$$\mathbb{𝔼}[U_{R}f(B)]=\mathbb{𝔼}[f(B)\mathbb{𝔼}[U_R\mid B]]=0.$$

Expanding the square gives

$$\begin{array}{cc}\hfill & \mathbb{𝔼}[(f+\alpha U_R-\mathrm{\Delta }{)}^2]-\mathbb{𝔼}[(f-\mathrm{\Delta }{)}^2]\hfill \\ \hfill & ={\alpha }^2\mathbb{𝔼}[U_R^2]+2\alpha \mathbb{𝔼}[U_R(f-\mathrm{\Delta })]\hfill \\ \hfill & ={\alpha }^2\mathrm{Var}(U_R)-2\alpha \mathbb{𝔼}[U_R\mathrm{\Delta }],\hfill \end{array}$$

where $\mathrm{\Delta }={\mathrm{\Delta }}_B^{\pi }(X)$. If $\mathbb{𝔼}[U_R\mathrm{\Delta }]\ne 0$, then $U_R$ is not almost surely zero, so $\mathrm{Var}(U_R)>0$ and this is a strictly convex quadratic in $\alpha$. Its minimizer is

$${\alpha }^{\star }=\frac{\mathbb{𝔼}[U_R\mathrm{\Delta }]}{\mathrm{Var}(U_R)},$$

and the reduction at that point is

$$\frac{\mathbb{𝔼}[U_R\mathrm{\Delta }{]}^2}{\mathrm{Var}(U_R)}>0.$$

If the numerator is positive, every sufficiently small positive $\alpha$ improves the corresponding binary-only target. Since $f^{\star }$ is the best binary-only target, taking $f=f^{\star }$ and $\alpha ={\alpha }^{\star }$ proves the dominance claim in Proposition 1.

A.2 Proof of Proposition 2

For a failed rollout $i$ in a mixed group, $B_i=0$ and $\bar{B}=s/G$. Also,

$$R_i-\bar{R}=\frac{G-1}{G}(R_i-{\bar{R}}_{-i}).$$

Therefore $A_i^{\mathrm{m}\mathrm{i}\mathrm{x}}>0$ exactly when

$$(1-\beta )\frac{G-1}{G}(R_i-{\bar{R}}_{-i})>\frac{\beta s}{G},$$

which rearranges to the stated condition.

For a terminal fractional return, every successful rollout has $R=1$, so ${\bar{R}}_{-i}\ge s/(G-1)$. If failures can score anywhere from 0 up to $r_{\max }$, the largest possible gap is $R_i-{\bar{R}}_{-i}=r_{\max }-s/(G-1)$, attained when rollout $i$ scores $r_{\max }$ and every other failure scores zero. Substituting it into the positive-label condition shows that a reversal is possible exactly when

$$\beta <1-\frac{s}{(G-1)r_{\max }}.$$

Across all verifiers, $r_{\max }$ can be arbitrarily close to one, which gives the bound $1-s/(G-1)$. Any smaller $\beta$ admits a reversal for some verifier, and no $\beta$ at or above it admits one for any verifier.

A.3 Proof of Proposition 3

In a mixed group, $0<s<G$. If rollout $i$ succeeds,

$$A_i^{\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{e}}=1-\frac{s}{G}=\frac{G-s}{G}>0.$$

If it fails,

$$A_i^{\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{e}}=0-\frac{s}{G}<0.$$

The fractional branch is not used in either calculation.

A.4 Proof of Proposition 4

Let $g$ denote a generic trajectory score, so $\mathbb{𝔼}[g]=0$. For any terminal scalar $Y$ with no explicit parameter dependence, inclusive group centering gives

$$\mathbb{𝔼}[\frac{1}{G}\sum _{i=1}^G(Y_i-\bar{Y})g_i]=c_G\nabla \mathbb{𝔼}[Y],$$

because $\mathbb{𝔼}[Y_i{g}_i]=\mathbb{𝔼}[Yg]=\nabla \mathbb{𝔼}[Y]$ by the score-function identity, the $j=i$ term of the group mean removes $G^{-1}\mathbb{𝔼}[Yg]$ from it, and the $j\ne i$ terms vanish by independence and $\mathbb{𝔼}[g]=0$.

Centered binary return is zero on all-pass and all-fail groups, so the binary-gated label can be written as $A_i^{\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{e}}=(B_i-\bar{B})+{\eta }_0\mathbf{𝟏}\{s=0\}(R_i-\bar{R})$. The first part is centered binary return, whose expected update is $c_G\nabla p$.

Now condition on the all-fail event. It has probability $q^G$, and conditional on that event the rollouts are independent draws from the policy conditioned on $B=0$. The conditional score $\mathbb{𝔼}[g\mid B=0]$ is not zero in general, so the identity above does not apply directly. Expanding the group mean instead, the diagonal terms give $G^{-1}\mathbb{𝔼}[Rg\mid B=0]$ and each off-diagonal term gives ${\mu }_0\mathbb{𝔼}[g\mid B=0]$ by conditional independence. Hence

$$\mathbb{𝔼}[\frac{1}{G}\sum _{i=1}^G(R_i-\bar{R})g_i|s=0]=c_G\mathbb{𝔼}[(R-{\mu }_0)g\mid B=0].$$

Since $\nabla q=\mathbb{𝔼}[\mathbf{𝟏}\{B=0\}g]=q\mathbb{𝔼}[g\mid B=0]$, differentiating ${\mu }_0=\mathbb{𝔼}[R\mathbf{𝟏}\{B=0\}]/q$ gives

$$\begin{array}{cc}\hfill \nabla {\mu }_0& =\mathbb{𝔼}[Rg\mid B=0]-{\mu }_0\frac{\nabla q}{q}\hfill \\ \hfill & =\mathbb{𝔼}[Rg\mid B=0]-{\mu }_0\mathbb{𝔼}[g\mid B=0]\hfill \\ \hfill & =\mathbb{𝔼}[(R-{\mu }_0)g\mid B=0].\hfill \end{array}$$

Multiplying by the all-fail probability $q^G$ and by ${\eta }_0$ yields the second term of Proposition 4.

The same centering identity with $Y=\beta B+(1-\beta )R$ gives the fixed-mixture update. Since $\mathbb{𝔼}[R]=p+q{\mu }_0$,

$$\nabla \mathbb{𝔼}[R]=(1-{\mu }_0)\nabla p+q\nabla {\mu }_0,$$

and therefore $\mathbb{𝔼}[{\hat{g}}_{\mathrm{m}\mathrm{i}\mathrm{x}}]=c_G[(1-(1-\beta ){\mu }_0)\nabla p+(1-\beta )q\nabla {\mu }_0]$.

A.5 Proof of Proposition 5

By A.4, $\mathbb{𝔼}[(R-{\mu }_0)g\mid B=0]=\nabla {\mu }_0$, so

$${\tau }_c=\frac{⟨\nabla {\mu }_0,v_1⟩}{{\Vert v_1\Vert }^2}.$$

Since $B$ is binary, $\nabla p=\mathbb{𝔼}[Bg]=p{v}_1$. Therefore

$$\frac{⟨\nabla p,\nabla {\mu }_0⟩}{{\Vert \nabla p\Vert }^2}=\frac{p⟨v_1,\nabla {\mu }_0⟩}{p^2{\Vert v_1\Vert }^2}=\frac{{\tau }_c}{p}.$$

Taking the inner product of Proposition 4 with $\nabla p$ and dividing by $c_G{\Vert \nabla p\Vert }^2$ gives

$$\frac{⟨\nabla p,\mathbb{𝔼}[{\hat{g}}_{\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{e}}]⟩}{c_G{\Vert \nabla p\Vert }^2}=1+{\eta }_0{q}^G\frac{{\tau }_c}{p}.$$

Binary GRPO has expected update $c_G\nabla p$, whose normalized projection is exactly $1$. With ${\eta }_0>0$, $q>0$, and $p>0$, the binary-gated projection exceeds binary GRPO’s exactly when ${\tau }_c>0$, and it is positive exactly when ${\tau }_c>-p/({\eta }_0{q}^G)$. The same computation applied to the fixed-mixture update in A.4 gives $1-(1-\beta ){\mu }_0+(1-\beta )q{\tau }_c/p$.

For a preconditioned update $P\hat{g}$ with $P$ symmetric positive definite, the same steps with inner products $⟨a,b{⟩}_P=a^{\top }Pb$ replace ${\tau }_c$ by $⟨\nabla {\mu }_0,P{v}_1⟩/⟨v_1,P{v}_1⟩$.

A.6 Proof of Proposition 6

Let $S={\sum }_{j=1}^M{\sigma }_j$, let

$$p=\prod _{j=1}^M{\sigma }_j,$$

and write $a_j={\prod }_{\ell \ne j}{\sigma }_{\ell }$, so $p={\sigma }_j{a}_j$. Since $R$ is the unweighted pass fraction,

$$\mathbb{𝔼}[R]=\frac{S}{M}.$$

Because $R=1$ on the all-pass event,

$${\mu }_0=\frac{\mathbb{𝔼}[R]-p}{1-p}=\frac{S/M-p}{1-p}.$$

Clearly

$$\frac{\partial p}{\partial {\sigma }_j}=a_j>0.$$

Differentiating ${\mu }_0$ gives

$$\frac{\partial {\mu }_0}{\partial {\sigma }_j}=\frac{1-a_j(M-\sum _{\ell \ne j}{\sigma }_{\ell })}{M(1-p{)}^2}.$$

Set $u_{\ell }=1-{\sigma }_{\ell }$ for $\ell \ne j$. The numerator is positive precisely when

$$\prod _{\ell \ne j}(1-u_{\ell })(1+\sum _{\ell \ne j}{u}_{\ell })<1.$$

This holds because

$$\prod _{\ell \ne j}(1-u_{\ell })\le e^{-\sum _{\ell \ne j}{u}_{\ell }}\le \frac{1}{1+\sum _{\ell \ne j}{u}_{\ell }},$$

and because $M\ge 2$ and every ${\sigma }_{\ell }$ lies in $(0,1)$, the final inequality is strict. Hence $\partial {\mu }_0/\partial {\sigma }_j>0$.

Let $J_{jk}=\partial {\sigma }_j/\partial {\theta }_k\ge 0$ denote the Jacobian of a parameterization in which every check probability is nondecreasing in every parameter. For every parameter coordinate $k$,

$$\frac{\partial p}{\partial {\theta }_k}=\sum _{j=1}^M\frac{\partial p}{\partial {\sigma }_j}{J}_{jk}\ge 0,\frac{\partial {\mu }_0}{\partial {\theta }_k}=\sum _{j=1}^M\frac{\partial {\mu }_0}{\partial {\sigma }_j}{J}_{jk}\ge 0.$$

Therefore,

$$⟨{\nabla }_{\theta }p,{\nabla }_{\theta }{\mu }_0⟩=\sum _k\frac{\partial p}{\partial {\theta }_k}\frac{\partial {\mu }_0}{\partial {\theta }_k}\ge 0.$$

If $\nabla p\ne 0$, some coordinate $k$ has $\partial p/\partial {\theta }_k>0$, which requires $J_{jk}>0$ for some $j$. For that $k$, $\partial {\mu }_0/\partial {\theta }_k\ge (\partial {\mu }_0/\partial {\sigma }_j)J_{jk}>0$, so the inner product is strictly positive. By A.5, ${\tau }_c=p⟨\nabla p,\nabla {\mu }_0⟩/{\Vert \nabla p\Vert }^2$, so ${\tau }_c$ has the same sign. This proves the result.

A.7 The three-strategy table

Write ${\pi }_j$ for the probability of strategy $j$, $b_j$ for its completion probability, and $r_j$ for the fractional return of its failures, so that $(b_j,r_j)$ is $(0.9,0.983)$, $(0,0.983)$ and $(0,0)$ for the three strategies. With logits $z_j={\kappa }_j{\varphi }_j$ and gradients taken with respect to $\varphi$, the softmax identity $\partial {\pi }_j/\partial z_k={\pi }_k({\delta }_{jk}-{\pi }_j)$ gives, for any per-strategy quantity $y_j$ with mean $\bar{y}={\sum }_j{\pi }_j{y}_j$,

$$\frac{\partial \bar{y}}{\partial {\varphi }_k}={\kappa }_k{\pi }_k(y_k-\bar{y}).$$

Apply this with $y_j=b_j$, whose mean is $p$, and with $y_j=(1-b_j)r_j$, whose mean is $\nu =\mathbb{𝔼}[R\mathbf{𝟏}\{B=0\}]=q{\mu }_0$. Since ${\mu }_0=\nu /q$ and $\nabla q=-\nabla p$,

$$\nabla {\mu }_0=\frac{\nabla \nu +{\mu }_0\nabla p}{q},$$

and by A.5, ${\tau }_c=p⟨\nabla p,\nabla {\mu }_0⟩/{\Vert \nabla p\Vert }^2$. With $\pi =(0.02,0.49,0.49)$, these formulas give $p=0.018$, ${\mu }_0=0.4925$ and the ${\tau }_c$ column of the table. The progress column is $1+{\eta }_0{q}^G{\tau }_c/p$ with $G=8$ and ${\eta }_0=0.5$.

A.8 The binary-anchored label

In a mixed group, both outcome classes are nonempty. For $b\in \{0,1\}$, let ${\bar{R}}_b$ be the mean fractional return of the rollouts with $B_j=b$, and let $d_i=R_i-{\bar{R}}_{B_i}$ be the deviation of rollout $i$ from the mean of its own class. The binary-anchored label is

$$A_i^{\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{h}\mathrm{o}\mathrm{r}}=\{\begin{array}{cc}(B_i-\bar{B})[1+\lambda (2B_i-1)d_i],\hfill & 0<s<G,\hfill \\ {\eta }_0(R_i-\bar{R}),\hfill & s=0,\hfill \\ 0,\hfill & s=G,\hfill \end{array}0\le \lambda <1.$$

It differs from the binary-gated label only in mixed groups. There it assigns a strictly positive label to every successful rollout and a strictly negative label to every failed rollout, and its labels sum to zero.

Because $R\in [0,1]$, every deviation satisfies $d_i\in [-1,1]$, so

$$1+\lambda (2B_i-1)d_i\ge 1-\lambda >0.$$

Each label therefore has the sign of $B_i-\bar{B}$, which A.3 shows is positive for successes and negative for failures. For the sum, ${\sum }_i(B_i-\bar{B})=0$, and within class $b$ the correction terms add up to $(b-\bar{B})\lambda (2b-1){\sum }_{i:B_i=b}{d}_i=0$, because deviations from a class mean sum to zero.

For a terminal fractional return, every success has $R=1$, so $d_i=0$ on successes and anchoring rescales only failures. A failed rollout receives $-(s/G)(1-\lambda d_i)$, so a failure that scores above the failure mean is pushed down less than one that scores below it. The expected mixed-group term is therefore generally not $c_G\nabla p$. It carries an additional term that, like the fallback, favors higher-scoring failures.