BPCO bounded critic: a critic that can only say a number the reward could be

​

Recipe

1. Base: Qwen/Qwen2.5-1.5B-Instruct. Data: GSM8K, 512 train prompts from the train split, 120 held out from the test split, decontaminate run against the holdout and the dropped count recorded. Reward, both arms: the binary outcome, wai.verify.MathEqual() against the GSM8K gold number. A program, not a judge.
2. The regime, both arms: one rollout per prompt, 64 prompts per optimizer step, 256 new tokens at most, temperature 1.0 with top-p 1.0, top-k off and no repetition penalty (Qwen’s generation config sets all three; a truncated sampler has no common support with the policy). The rollouts are drawn by a stale sampler: a frozen copy of the policy’s LoRA weights, refreshed only every LAG = 4 optimizer steps, so a batch was written by a policy 1 to 4 updates old (the first block, and every warm-up step, is on-policy: there is no older policy to copy). Each trajectory carries behavior_logprobs from the sampler (a teacher-forced pass under its weights, right after generation, the way verl takes its old log-probs) and logprobs from the current policy at update time (one teacher-forced pass), plus reward. Only generated tokens are actions.
3. The critic, both arms: one linear layer on the trunk’s last hidden state at the position that predicted each generated token (the same forward pass, hidden states detached, so the critic never trains the trunk), its own AdamW at 1e-4. The paper’s 1e-5 (BPCO_CRITIC_LEARNING_RATE) is for a full separate value model; a 1,537-parameter head on a frozen feature needs more, and 1e-4 is the override. Both arms’ heads share the same init (set_seed(17) right before the adapter and the head are built).
4. Baseline arm: the head’s raw output is the value; the critic trains toward lambda-returns (GAE lambda 0.95, gamma 1) with verl’s clipped value loss, clip 0.5; the advantages are whitened over the batch; the ratio against the rollout policy is clipped to [0.8, 1.2]; no warm-up. Written out in baseline_update: coefficients[i][t] is rho A where the unclipped branch of min(rho A, clip(rho) A) is the minimum and 0 where the clipped branch is, which is that surrogate’s exact gradient at one step per batch (the selftest checks it against autograd).
5. Recipe arm: the head’s raw output goes through wai.BPCO().bound(z), done in torch with the same formula so the gradient flows (the selftest checks it against bound() on 2,401 points), and trains toward update.value_targets, the Monte Carlo return, with MSE; critic_warmup = 15 critic-only steps first, read off the object; then the policy step with -(coef * logprob).sum() / N, the coefficients from wai.BPCO().update(batch) held constant (the 1/N is the batch mean the Update contract leaves to the trainer; the library’s own toy trainer applies the same). The baseline uses the same loss line with its own coefficients, so the two arms differ in who wrote the coefficients and the value targets, and in nothing else.
6. Training, both arms: LoRA rank 16, alpha 32 (TRAINING_LORA_RANK, TRAINING_LORA_ALPHA) on the seven projections, AdamW at 1e-5 on the adapter (the paper’s 1e-6, BPCO_LEARNING_RATE, is full weights), gradient clip 1.0, one step per batch, 40 policy steps after any warm-up, seed 17 set right before the trainer is built. Logged per step: mean reward, clipped token share, mean ratio, mean advantage, explained variance against the Monte Carlo return in both arms (equation 10 of the paper), completion length; the series are in results.json under each arm’s curve.
7. Eval: pass@1 on the same 120 held-out tasks, 4 samples per task at temperature 0.7 (same top-p, top-k and penalty settings as training). The untrained base is evaluated three times first, and that spread is the noise floor a delta has to clear. Paired delta with a 95% interval (wai.pass_at, wai.compare with run_std, run_std_runs, train_runs and proxy= naming the training reward). hack_scan on the last training batch of each arm.

​

Run

python recipe.py --selftest   # the parser, the sampler bookkeeping, the bound, both update rules, the assembly; offline, no GPU and no key
python recipe.py              # both arms, one container each, sized for under 60 GPU minutes on one L40S
python recipe.py --arm recipe --steps 3 --critic-warmup 2 --n-holdout 16 --base-runs 1   # a few cents, proves the loop

​

Result

​

Checks

​

Climb

​

Learned

• The regime the paper is about was not reached, and the clip counters say so. With the adapter at 1e-5 the policy moved so little in 40 single-rollout steps that a sampler four updates stale was still within 1 -/+ 0.0014 in ratio; PPO’s clip touched 0.17% of tokens, DPPO’s 0.001%. Two of the bundle’s five parts (the clip and the warm-up before the policy moves) cannot change a gradient in that regime, so what this run compares is the bounded critic and its target and advantage against the unbounded one, and on pass@1 that comparison is flat (-0.017 [-0.069, +0.037]). A learning rate the adapter can move at (1e-4, where adaptive-clip’s GRPO went from 0.34 to 0.47 in the same 40 steps) is the knob that would put the run where the paper’s claim lives; it is a deliberate deviation from the shared single-rollout protocol and belongs in a round of its own.
• Bounding the critic made it wrong by less, not right. The unbounded head’s explained variance started at -15 (its raw outputs were far outside the reward range, which is the paper’s complaint in one number) and was still -4.9 after 40 steps of clipped value loss; the bounded head started at -0.36 and ended at -0.24. Neither beat the batch mean, so neither arm’s advantage carried much beyond the reward itself. A linear head on a frozen feature at 1e-4, 64 sequences a step, is a weaker critic than the paper’s separate value model, and the 15-step warm-up moved its loss from 0.37 to 0.32 without moving its explained variance. What would test the critic’s part of the claim is a critic that learns: a longer warm-up, or a small trainable projection under the head.
• Length is what moved, in both arms, with the reward flat. Rollouts shortened by a quarter in the baseline and by nearly half in the recipe, eval answers from 729 to 592 and 284 characters, and pass@4 rose while pass^4 fell in the recipe arm. With a critic near 0.5 and a binary reward, every token of a wrong answer carries a negative residual and a short answer carries fewer of them; the unnormalized LA-GAE advantage does not subtract that off the way whitening does, which is one reading of why the recipe arm shortened faster. That is a hypothesis this run cannot separate from the learning rate, and a second seed per arm (train_runs=) comes before any of it: at one seed the verdict is unresolved and stays so.

​

References

1. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter Policy Gradient Algorithms.
2. Qi, P., Zhou, X., Lee, W. S. Best Practice Critic Optimization. arXiv:2608.23566, 2026.
3. Schulman, J. et al. Proximal Policy Optimization Algorithms. arXiv:1707.06347, 2017.
4. Schulman, J. et al. High-Dimensional Continuous Control Using Generalized Advantage Estimation. arXiv:1506.02438, 2016.
5. Sheng, G. et al. HybridFlow: A Flexible and Efficient RLHF Framework. arXiv:2409.19256, 2024.
6. Qi, P. et al. Rethinking the Trust Region in LLM Reinforcement Learning. arXiv:2602.04879, 2026.
7. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapter Evaluation.
8. Lambert, N. et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124, 2024.
9. Gao, L., Schulman, J., Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML 2023. arXiv:2210.10760.