Skip to main content
The scripts are in the repository, not in the installed package. Clone it, then cd recipes/papers/zero-rl-format-reward before running the commands below. Browse this recipe on GitHub.
Paper: SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild, Weihao Zeng et al., arXiv:2503.18892, March 2025. https://arxiv.org/abs/2503.18892 Book: RL with a verifiable reward, a program that checks the answer [1], run straight on a base model with no SFT first [2], and what the reward is allowed to depend on. Claim: A base model trained with GRPO and a rule reward gains accuracy on math without any SFT first, but only if the reward pays for the answer and not the format; requiring \boxed{} and punishing its absence “penalizes many correct explorations” early and shrinks the chain of thought late (section 3.1). The change: the baseline arm is paid the R1-style strict reward (+1 for the gold inside the last \boxed{}, -1 with no box, 0 otherwise); the recipe arm is paid +1 for a correct answer however it is written and 0 otherwise, the paper’s default. Nothing else differs.

​
Recipe

  1. Base: Qwen/Qwen3.5-4B-Base, a hybrid model (24 gated DeltaNet blocks, 8 full-attention blocks), no chat turns, plain-text prompt. Data: MATH train at levels 3 to 5 (the paper’s “Hard” tier, the one it pairs with Qwen-class bases), 1,024 train prompts; MATH-500, 160 held out, drawn from the MATH test split so the two are disjoint by construction.
  2. Baseline: TRL 1.13 GRPOTrainer with a LoRA adapter (rank 32, every linear layer), rollouts from a colocated vLLM engine, 30 steps of 6 prompts x 8 rollouts, up to 1,536 completion tokens, lr 5e-5, clip 0.2, KL 1e-4 (the paper’s coefficient for models up to 14B), temperature 1.0, loss_type="dapo" [3] (the paper removes GRPO’s length normalisation, appendix B.5). Reward: strict.
  3. Recipe: the same run, reward lenient.
  4. Eval: pass@1 on the same 160 held-out tasks, 4 samples per task, read leniently for both arms (MathEqual on \boxed{}, “the answer is”, or the last line), so the target is one number. The untrained base is evaluated three times first and that spread is the noise floor; the train set is decontaminated against the holdout before training. Paired delta with a 95% interval (wai.pass_at, wai.delta_report).
  5. delta_report(proxy="strict_reward"): the baseline trained on a reward that is not the target metric, and the report says whether it over-optimised it.
The paper runs 8,192-token rollouts with 1,024 prompts a batch on the full model; this recipe cuts the length to 1,536, the batch to 6 prompts and the update to a LoRA adapter so both arms fit in under an hour on one H100. Those cuts are the same for both arms.

​
Run

​
Result

Run 2026-09-18, both arms, one H100, 46.2 GPU minutes, $3.08. Recipe vs baseline: +0.094 [+0.052, +0.139] over 160 paired tasks. Verdict: moved. The interval excludes zero and the delta is above the 0.067 noise band the three base re-runs set. Both arms beat the base. The strict arm gained 12 points in 30 steps, the lenient arm 21. The paper’s finding is the gap between them, and the gap survives its own noise floor. The proxy row is the other half of the finding. On the held-out set the strict reward itself went from 0.58 (baseline) to 0.53 (recipe), -0.052 [-0.119, +0.014], noise: the lenient arm boxes less (83% of replies against 96%) and is right more. The baseline trained toward the box and got it; the box was not the thing. Where the accuracy came from is in the training curves. The strict arm’s rollouts got shorter across the run (664 tokens over the first ten steps, 619 over the last ten; 15% then 6% at the cap) and its boxed share on the holdout rose from 0.91 to 0.96. The lenient arm’s rollouts stayed long (916 then 914 tokens, a third at the cap) and its held-out replies are 44% longer in characters (1,928 against 1,336). Under the strict reward a derivation cut off at the cap has no box and scores -1, the worst outcome available, so the policy learns to close early; under the lenient reward an unfinished derivation scores 0, the same as a wrong one, and the policy is free to keep thinking.

​
Checks

Nothing in this table is ticked by hand: every cell is written by recipe.py into results.json. Training-curve numbers above are from the trainer’s own log (completions/mean_length, completions/clipped_ratio, reward, kl), which the run page carries as the reward, KL and length curves. Zero-variance groups (frac_reward_zero_std) averaged 0.29 on the strict arm and 0.41 on the lenient arm: at a 0.57 mean reward the lenient arm wastes more of its groups on all-correct prompts, which is what DAPO’s dynamic sampling [3] is for and the next knob to turn.

​
Climb

Round 1 died on the recipe arm’s first loss forward: TRL upcasts the full 248k-token-vocabulary logits to fp32 before the per-token log-probabilities, and eight 1.5k-token sequences of that is 11 GB the H100 no longer had next to the colocated engine. The strict arm had survived the same forward at the edge. Its result was lost with the process, because results.json was written at the end; rows now land in .cache/<arm>.json as each arm returns and --reuse rebuilds the delta from disk. The strict arm scored 0.63 in both rounds, by two different paths: in round 1 its rollouts ran away to 1,400 tokens with 81% at the cap by step 12 (the paper’s Figure 6 picture), in round 2 they stayed near 620. Same held-out number, so the length runaway is a training-seed event and not what the score is made of; more seeds per arm is the next thing to buy before this delta is banked.

​
Learned

  • The gap is real at this size and it is not about the box. Base replies already box 91% of the time, so the -1 penalty rarely fires on the holdout; it fires during training on every rollout that runs out of room, and that is enough to change what the policy learns to do with its budget.
  • Score the target, not the reward. delta_report(proxy="strict_reward") shows the lenient arm losing on the baseline’s own reward while winning on accuracy; without that row the baseline’s 96% boxed share would look like the better-trained model.
  • Qwen3.5’s hybrid architecture trains under TRL 1.13 with vLLM colocated and LoRA on every linear layer, at 70 to 85 s a step for 48 rollouts of 1.5k tokens on one H100. The memory ceiling is the fp32 logits in the loss forward, not the engine; keep the per-device batch at half a group.
  • Next: three seeds per arm, and DAPO’s dynamic sampling [3] on the lenient arm, whose groups saturate at a 0.57 mean reward.
Verified 2026-09-18, whileai 0.83, TRL 1.13.0 GRPOTrainer + vLLM 0.29.0 on Modal. Run page: https://withwhile.com/platform/training/run_064f9185fdf88eef (recipe arm), https://withwhile.com/platform/training/run_42f97ae9878c4621 (baseline arm); Runs page: https://while.ai/platform/runs?agent=zero-rl-qwen3.5-4b

​
References

  1. Lambert, N. et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124, 2024.
  2. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter Reasoning.
  3. Yu, Q. et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476, 2025.
  4. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter Evaluation.
  5. Gao, L., Schulman, J., Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML 2023. arXiv:2210.10760.
Last modified on September 19, 2026