> ## Documentation Index
> Fetch the complete documentation index at: https://docs.withwhile.com/llms.txt
> Use this file to discover all available pages before exploring further.
> ## Agent Instructions
> Install with `uv add whileai`; import as `import whileai as wai`.
> Run the offline path first (`simulator=False`, `wai.seeded_agent`, a callable judge); no key is needed for it.
> Report every pass rate with its interval and n, as `scored.pass_at` prints it.
# Process reward or outcome reward: does the ten-point gap survive?
> arXiv:2607.02869 (Palandye et al., July 2026) trains Qwen2.5-0.5B with GRPO on GSM8K under five reward regimes and reports that scoring the steps beats scoring the answer: 63.73% test accuracy for process-only against 5.
The scripts are in the repository, not in the installed package. Clone it,
then `cd recipes/community/process-vs-outcome-reward` before running the commands below. [Browse this recipe on GitHub](https://github.com/whilehq/whileai-sdk/tree/main/recipes/community/process-vs-outcome-reward).
arXiv:2607.02869 (Palandye et al., July 2026) trains Qwen2.5-0.5B with GRPO
on GSM8K under five reward regimes and reports that scoring the *steps*
beats scoring the *answer*: **63.73%** test accuracy for process-only
against **53.75%** for outcome-only. Single runs, no intervals, no seeds.
This recipe asks the smaller question an outsider can afford: with one hour
of one GPU and the same one change, does the gap appear at all, and is it
bigger than the noise?
What you will learn: how to score a public benchmark with whileai's
measurement calls, how `eval_variance` turns three base passes into a band
that a delta has to clear, and what `compare(proxy=...)` says when the
reward you trained on is not the metric you care about. You need a Modal
account for the two arms; `--dry-run` needs nothing.
## The one change
Everything is held fixed between arms except the reward the group is
scored with. Both are programs, not judges.
| arm | reward |
| --------- | ----------------------------------------------------------------------------------------------------------- |
| `outcome` | `MathEqual` on the final answer — whileai's verifier, unmodified |
| `process` | correct steps / total steps, matched to 1e-5, scaled down when the chain runs past 1.5x the gold step count |
`rewards.py` writes the process reward because the paper does not say how a
"step" comes out of a free-form completion, nor how big the length penalty
is. Both choices are at the point they are made, and `--dry-run` prints the
table that shows the two rewards genuinely come apart:
```text theme={"theme":"vitesse-dark"}
completion R_process R_outcome
the gold chain 1.000 1.000
right steps, wrong answer 1.000 0.000
right answer, no work 0.500 1.000
padded chain, right answer 0.250 1.000
```
## Run it
```bash theme={"theme":"vitesse-dark"}
uv add whileai
cd recipes/community/process-vs-outcome-reward
python run.py --dry-run # offline: the reward programs, no key, no GPU
python run.py prep # held-out set + decontamination -> prompts.json
modal run train_modal.py # both arms, one L40S each, in parallel
modal volume get wai-seat5-out '**' out/
python run.py analyze # the numbers below
modal app stop wai-seat5-granularity
```
| flag | default | what it does |
| ----------- | ------- | ----------------------------------------------------------- |
| `--dry-run` | off | the reward programs on a built-in row; no network, no key |
| `stage` | `prep` | `prep` writes `prompts.json`, `analyze` reads the arms back |
| `--out` | `out` | where `modal volume get` put the two arm files |
## What you get
Both rewards work. The gap between them does not survive its own interval.
Qwen2.5-0.5B-Instruct, 192 GSM8K training prompts (one epoch, 48 GRPO steps,
5 rollouts each, LoRA r=16), 200 held-out GSM8K test questions at k=2,
sampled at T=0.7. Held-out pass\@1, paired, 95% interval from a task
bootstrap:
| arm | held-out pass\@1 | delta vs base | verdict |
| ----------------- | --------------------- | ---------------------------- | --------------------------- |
| untrained base | 0.273 / 0.258 / 0.263 | — | noise floor, run\_std 0.008 |
| outcome-only | **0.325** | +0.052 \[+0.010, +0.092] | moved |
| process-only | **0.355** | +0.083 \[+0.025, +0.140] | moved |
| process − outcome | — | **+0.030 \[−0.022, +0.085]** | **flat** |
The paper reports process-only beating outcome-only by **+9.98 points**
(63.73 vs 53.75). Here the same one change is worth **+3.0 points**, with an
interval that contains zero and whose upper bound (+8.5) sits just under the
paper's number. The verdict rule is the interval, not the sign: a delta has
to clear zero *and* clear the noise band, and `compare()` puts that band at
0.046 from three base passes. +0.030 clears neither.
> **Which noise band.** 0.046, not 0.021. `run_std` estimated from three
> re-runs is an estimate, not the eval's exact spread, so the quantile is
> t(df=2)=4.30 rather than 1.96 — and the difference decides this run, since
> +0.030 clears the 1.96 form and not the real one. When this run was made,
> `eval_variance()["noise_band"]` returned the 1.96 form while `compare()`
> applied the t one, which is [#616](https://github.com/whilehq/whileai-sdk/issues/616);
> since that fix `eval_variance` carries the band `compare()` applies and
> says so with `noise_band_df`, and `run.py` reads it off the report. The
> number judged against was 0.046 throughout, so nothing here moved.
> `results.json` is the record of the run as it happened, so it still carries
> both values under `noise_band_eval_variance` and `noise_band_applied`; a
> re-run on the fixed library writes `noise_band_applied` and
> `noise_band_df`.
Both arms beat the untrained base, which is the part that did replicate.
`compare(proxy="marker:process")` also flags the arm comparison
**OVER-OPTIMIZED**: the process metric moved +0.084 \[+0.045, +0.122] while
final-answer accuracy moved +0.030 inside the noise. Training on the steps
moves the steps a lot more than it moves the answer.
### The mechanism is visible even though the result is flat
From the two training logs, on the same rollout budget:
| | process | outcome |
| --------------------------------------------------- | -------------- | -------------- |
| groups with zero advantage (`frac_reward_zero_std`) | **0.083** | **0.370** |
| completions that hit the 256-token cap | 0.156 | 0.317 |
| KL from base at the last step | 0.015 | 0.005 |
| held-out reply length after training (chars) | 478 (from 716) | 637 (from 716) |
This is the paper's argument, measured: a binary outcome reward leaves
**37%** of GRPO groups with all five rollouts scoring the same, so those
groups produce no gradient at all. The dense process reward wastes 8%. That
is 4.5x more usable groups per step — it simply has not turned into a
separable accuracy difference by step 48.
### Did either reward get gamed?
`hack_scan(endorsed=["marker:process"])` on each trained arm's held-out
rollouts, with `numbers_emitted` added by hand because a process reward
invites exactly that cheat — pad the chain with numbers and hope one matches
a gold step:
| | process | outcome |
| --------------------------------------------------------------- | ---------------- | ---------------- |
| top feature | `marker:process` | `marker:process` |
| integrity (above-floor signal on an endorsed feature) | **1.00** | **1.00** |
| gradient capacity (asks that are neither all-pass nor all-fail) | 0.25 | 0.24 |
Nothing above the noise floor except the endorsed feature: not length, not
truncation, not `numbers_emitted`. The scan also reaches the dead-group
finding from the eval side — only a quarter of held-out asks carry gradient
at all — which is the same story the training logs tell. Its own warning
applies: at k=2 the floor is coarse, and it asks for k>=8 before acting on a
close call.
## Cost
Two L40S containers, \~20 minutes each including image build and model
download, run in parallel: **\~40 L40S-minutes, about \$1.30** at Modal's
posted L40S rate. No model API key was used anywhere in this recipe; the
reward is a program and the eval is the same program.
## What did not work
* **The paper does not define its own process reward well enough to
reimplement.** "Correct steps / total steps" leaves open what a step is in
a free-form completion, and "a penalty when generated chains exceed 1.5x
the ground-truth step count" gives no magnitude. `rewards.py` counts gold
steps from GSM8K's `<>` annotations, matches candidate numbers as a
multiset to 1e-5, and scales by `1.5 * gold / candidate` past the
threshold. A different reading of the same two sentences is a different
experiment, and that is the largest single source of disagreement between
this number and the paper's.
* **My first process reward scored the gold solution 0.27.** It counted every
number in the completion as a step, so the length penalty fired on correct
chains. Checking the reward against the reference answer before spending
any GPU time caught it; `--dry-run` is that check, kept.
* **This holdout cannot settle the paper's claim.** `holdout_size(0.10,
base=0.35, k=2)` asks for 187 paired tasks to prove a 10-point gain and
this recipe uses 200, which is enough for the paper's effect but not for
the effect actually observed: `compare()` reports that proving +0.030
would take about 1956 tasks. A flat verdict here is "not shown at this
scale", not "shown absent".
* **One epoch on 192 prompts is not the paper's run.** The paper does 5
epochs over 7,473 prompts with a full fine-tune at lr 1e-6 on VERL+vLLM;
this is one epoch over 192 with LoRA at lr 2e-5 on TRL. The base is
Qwen2.5-0.5B-**Instruct** rather than the base model, and eval is sampled
at T=0.7 rather than greedy, so that a noise floor exists at all.
* **The three base passes were not three draws.** Same seeds, same model and
same batching give byte-identical passes, so the second container
reproduced the first's `[0.2725, 0.2575, 0.2625]` exactly. The spread here
is sampling noise across seeds, which is what the band needs; it is not
container-to-container variance, and this recipe cannot speak to that.
## Next
`python run.py analyze` writes `results.json`. To push further: raise
`N_TRAIN` past one epoch of 192 prompts, or add the paper's three hybrid
arms (`Weighted([(process, l), (outcome, 1 - l)])` is already in the SDK)
and see whether l = 0.9 sits where the paper puts it.