Train on what the model gets right sometimes

Not every row is worth training on. For SFT the rule is simple: keep the rows that passed, because those are the examples to copy. For RL the rule is stranger, and it is the one idea in this course that surprises people. A task the model always gets right teaches it nothing, because there is nothing to fix. A task it always gets wrong teaches it nothing either, because there is no good try to push toward. RL learns from the difference between tries of the same task. So you keep the tasks in the middle.

The mechanism

Here is the RL update in plain words, for the method most teams use today, GRPO. Take one task and its four tries. Score each. Subtract the group’s average score from each try. The tries above average get pushed up, the ones below get pushed down. If all four scored the same, every difference is zero and the update does nothing. A group like that is a unanimous group, and the library drops it before it reaches the trainer. The rule of thumb that follows is the 20 to 80 percent band: keep tasks the model currently passes between one time in five and four times in five. Below that it cannot learn yet. Above it, it already knows. Two more gates run at the same time.

Duplicates. The stand-in agent repeats itself, and so do real agents at low temperature. Identical tries carry no difference to learn from.
What the reward is really tracking. A model learns whatever gets the score. If longer replies happen to score higher, it learns to be long. The scan checks every reward against features like length and hedging, within each task, and warns when one predicts the reward. That warning is a reason to look at the judge before you train, not a reason to skip it.

Run it

import whileai as wai


@wai.tool
def get_order(order_id: str) -> dict:
    """Look up an order by id."""
    ...


data = wai.simulate(
    wai.seeded_agent([get_order]),
    tools=[get_order],
    system_prompt="Help customers with orders.",
    simulator=False,
    mode="rl",
    repeats=4,
    repeat_policy="fixed",
    budget=64,
    seed=0,
)
scored = data.grade(judge=lambda row: {"reward": int(not row["seeded"])})

print(scored.select(mode="sft"))  # the passes
print(scored.select(mode="rl"))  # the middle

sft selection: kept 16 of 64 rows
  eligible 16 (reward >= 1.0), junk 0, not passing 21
  distinct behaviors: 6, covered: 6
  at most 4 completion(s) per prompt; rejection-sampling selection wants 10 to 30 so the pick is not biased (rlhfbook.com/c/10-rejection-sampling.html; Llama 3 samples 10 to 30). Raise repeats= if you mean to choose among completions rather than filter.
rl selection: kept 29 of 64 rows
  band 20%..80% pass rate: 0 asks dropped (0 too easy, 0 too hard)
  unanimous groups dropped: 5; duplicates dropped: 27; truncated drop: 0
  groups kept: 11
  hack scan: train
  warning: 27 duplicate rollout(s) within 13 ask(s) dropped
  warning: reward punishes reply length (corr -0.54); check the judge before training
  warning: reward punishes boilerplate (corr -0.31); check the judge before training
  warning: reward punishes hedging (corr -0.36); check the judge before training
  warning: 2 rollouts per ask at the median; the floor is coarse below 4, re-scan at repeats>=8 before acting on a close call
  warning: Difficulty was measured from 4 rollouts per task, so a task's band assignment can be off by about ±0.3. Use repeats=16 for a firmer band (the count the 20-80 band is measured from, rlhfbook.com/c/07-reasoning).
  warning: pass^k / pass@k do not survive the prune: the graded rows scored k=4, the selection leaves 2 rollout(s) per ask, so pass_at on these rows reports them as n/a. pass@1 and the carried calibration stamp still hold the graded measurement; take the k-way numbers from pass_at before optimize

The warnings are the point of the call, so read three of them.

unanimous groups dropped: 5. Five asks where all four tries scored the same. Nothing to learn there.
reward punishes reply length (corr -0.54). Here the judge reads the answer key, and the planted mistakes add words (an apology, a hedge), so shorter replies really are the better ones. On a real judge this line means: check whether it is grading the job or the word count.
Use repeats=16 for a firmer band. Four tries is a coarse estimate of a task’s pass rate. The band is measured from sixteen in the source it cites. Sixty-four rows is a lesson, not a training set.

Where it comes from

Shao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024. GRPO: the group average as the baseline.
Yu, Q. et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476, 2025. Dropping unanimous groups while sampling.
Yuan, Z. et al. Scaling Relationship on Learning Mathematical Reasoning with Large Language Models. arXiv:2308.01825, 2023. Keeping the passes for SFT, called rejection sampling.
Gao, L., Schulman, J., Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML, 2023. arXiv:2210.10760. Why a model learns what the score rewards rather than what you meant.
Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025. Chapters Rejection Sampling, Reasoning and Inference-Time Scaling (the 20 to 80 band) and Over-Optimization.

Training is done when the held-out score moved: export, train, and prove it.

From zero

Train on what the model gets right sometimes

The mechanism

Run it

Where it comes from

Next

​The mechanism

​Run it

​Where it comes from

​Next

The mechanism

Run it

Where it comes from

Next