Adaptive clip: the upper bound follows how rare a correct answer was

​

Recipe

1. Base: Qwen/Qwen2.5-1.5B-Instruct. Data: GSM8K, 512 train prompts from the train split, 120 held out from the test split (different splits, so there is no overlap to check for).
2. Reward, both arms: the binary outcome, MathEqual against the GSM8K gold number. A program, not a judge. The paper changes the clip, not the reward, so nothing here is shaped.
3. Baseline arm: GRPO with epsilon 0.20 and epsilon_high 0.28, fixed for every rollout. That pair is DAPO’s clip-higher [4] and it is the paper’s own token-level default.
4. Recipe arm: same 0.20 floor and the same 0.28 ceiling, but the upper bound slides per group, eps_hi(c) = eps_lo + (eps_hi_max - eps_lo) * (k - c) / (k - 1) with c correct out of k = 8 rollouts. One right out of eight keeps the full 0.28; seven right gets 0.2114.
5. Eval: pass@1 on the same 120 held-out tasks, 4 samples per task. The untrained base is evaluated three times first, and that spread is the noise floor a delta has to clear; the train set is decontaminated against the holdout before any training. Paired delta with a 95% interval (wai.pass_at, wai.delta_report).

​

Run

python recipe.py --selftest   # the clip schedule and the clamp, offline, no GPU and no key
python recipe.py              # both arms, sized for under 60 GPU minutes on one L40S
python recipe.py --arm recipe --eps-high-max 0.36

​

Result

​

Checks

​

Climb

​

Learned

• The bound can be made per-group without touching TRL’s loss body. epsilon_high is read inside _compute_loss, and torch.clamp takes tensor bounds, so handing it a (batch, 1) tensor broadcasts over the (batch, tokens) ratio and the clip_ratio metric TRL logs stays correct.
• Reading the group’s correct count off the sign of the advantage only works because this reward is binary: with rewards in {0, 1} the advantage is r - c/k, positive for exactly the correct rollouts. A shaped reward would break that and need the counts carried separately.
• Holding the ceiling equal across the arms is the honest comparison but it is also the conservative one: it makes the recipe a strictly tighter clip than the baseline. The paper compares against a fixed bound too, on a much bigger batch (256 prompts against 6 here), so a flat result could mean the batch rather than the idea.
• Check that your change is reachable before you spend a GPU hour on it. The per-group bound was implemented correctly, tested on the CPU against real TRL, and verified to arrive at the loss intact — and still could not move a gradient, because nothing in an on-policy run ever asks what the upper bound is. One line of the trainer’s own logging (clip_ratio/high_mean) would have said so before the run. It is now in the Checks a reader can see.
• GRPO was the intervention that mattered here. Both arms moved the base from 0.34 to 0.47-0.57 on GSM8K in 40 steps, which dwarfs everything the paper’s change could have done at this scale. A third of groups were flat on average today (frac_reward_zero_std mean 0.30, peaking at 0.67), so most of that came from the rollouts that split.
• A two-arm delta at one training seed per arm is not a measurement, and this is what that looks like. Re-running this recipe unchanged moved the delta from -0.065 [-0.117, -0.013] to +0.050 [0.000, 0.100]: 11.5 points, sign flipped, and the first run’s interval excluded zero on the wrong side. Nothing in the recipe changed; the differences were whileai 0.53 -> 0.64, ordinary nondeterminism in generation and kernel scheduling, and, on the baseline arm only, an unseeded LoRA init (TRL builds the adapter before it seeds; fixed in recipe.py after this round). The eval-noise floor this directory enforces (0.024 here) is the noise of re-running the eval on a fixed model, and it says nothing about the noise of re-running the training. Where a recipe’s whole claim is a delta between two trained models, that second source is the one that decides whether there is a result, and it needs several training seeds per arm to see at all. Read every one-seed delta in this directory — including the ones with tight intervals — with that in mind.

​

References

1. Schulman, J. et al. Proximal Policy Optimization Algorithms. arXiv:1707.06347, 2017.
2. Shao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024.
3. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter Reinforcement Learning.
4. Yu, Q. et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476, 2025.
5. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter Evaluation.
6. Lambert, N. et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124, 2024.
7. Gao, L., Schulman, J., Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML 2023. arXiv:2210.10760.