Process reward or outcome reward: does the ten-point gap survive?

​

The one change

  completion                    R_process  R_outcome
  the gold chain                    1.000      1.000
  right steps, wrong answer         1.000      0.000
  right answer, no work             0.500      1.000
  padded chain, right answer        0.250      1.000

​

Run it

uv add whileai
cd recipes/community/process-vs-outcome-reward
python run.py --dry-run           # offline: the reward programs, no key, no GPU
python run.py prep                # held-out set + decontamination -> prompts.json
modal run train_modal.py          # both arms, one L40S each, in parallel
modal volume get wai-seat5-out '**' out/
python run.py analyze             # the numbers below
modal app stop wai-seat5-granularity

​

What you get

Which noise band. 0.046, not 0.021. run_std estimated from three re-runs is an estimate, not the eval’s exact spread, so the quantile is t(df=2)=4.30 rather than 1.96 — and the difference decides this run, since +0.030 clears the 1.96 form and not the real one. When this run was made, eval_variance()["noise_band"] returned the 1.96 form while compare() applied the t one, which is #616; since that fix eval_variance carries the band compare() applies and says so with noise_band_df, and run.py reads it off the report. The number judged against was 0.046 throughout, so nothing here moved. results.json is the record of the run as it happened, so it still carries both values under noise_band_eval_variance and noise_band_applied; a re-run on the fixed library writes noise_band_applied and noise_band_df.

​

The mechanism is visible even though the result is flat

​

Did either reward get gamed?

​

Cost

​

What did not work

• The paper does not define its own process reward well enough to reimplement. “Correct steps / total steps” leaves open what a step is in a free-form completion, and “a penalty when generated chains exceed 1.5x the ground-truth step count” gives no magnitude. rewards.py counts gold steps from GSM8K’s <<a+b=c>> annotations, matches candidate numbers as a multiset to 1e-5, and scales by 1.5 * gold / candidate past the threshold. A different reading of the same two sentences is a different experiment, and that is the largest single source of disagreement between this number and the paper’s.
• My first process reward scored the gold solution 0.27. It counted every number in the completion as a step, so the length penalty fired on correct chains. Checking the reward against the reference answer before spending any GPU time caught it; --dry-run is that check, kept.
• This holdout cannot settle the paper’s claim. holdout_size(0.10, base=0.35, k=2) asks for 187 paired tasks to prove a 10-point gain and this recipe uses 200, which is enough for the paper’s effect but not for the effect actually observed: compare() reports that proving +0.030 would take about 1956 tasks. A flat verdict here is “not shown at this scale”, not “shown absent”.
• One epoch on 192 prompts is not the paper’s run. The paper does 5 epochs over 7,473 prompts with a full fine-tune at lr 1e-6 on VERL+vLLM; this is one epoch over 192 with LoRA at lr 2e-5 on TRL. The base is Qwen2.5-0.5B-Instruct rather than the base model, and eval is sampled at T=0.7 rather than greedy, so that a noise floor exists at all.
• The three base passes were not three draws. Same seeds, same model and same batching give byte-identical passes, so the second container reproduced the first’s [0.2725, 0.2575, 0.2625] exactly. The spread here is sampling noise across seeds, which is what the band needs; it is not container-to-container variance, and this recipe cannot speak to that.

​

Next