> ## Documentation Index > Fetch the complete documentation index at: https://docs.withwhile.com/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > Install with `pip install whileai`; import as `import whileai as wai`. > Run the offline path first (`simulator=False`, `wai.seeded_agent`, a callable judge); no key is needed for it. > Report every pass rate with its interval and n, as `scored.pass_at` prints it. # Train on what the model gets right sometimes > Lesson 6. Keep the passes for SFT. Keep the middle for RL, because a task the model always passes or always fails teaches nothing. Read the warnings. Not every row is worth training on. For SFT the rule is simple: keep the rows that passed, because those are the examples to copy. For RL the rule is stranger, and it is the one idea in this course that surprises people. A task the model always gets right teaches it nothing, because there is nothing to fix. A task it always gets wrong teaches it nothing either, because there is no good try to push toward. RL learns from the difference between tries of the same task. So you keep the tasks in the middle. ## The mechanism Here is the RL update in plain words, for the method most teams use today, **GRPO**. Take one task and its four tries. Score each. Subtract the group's average score from each try. The tries above average get pushed up, the ones below get pushed down. If all four scored the same, every difference is zero and the update does nothing. A group like that is a **unanimous group**, and the library drops it before it reaches the trainer. The rule of thumb that follows is the **20 to 80 percent band**: keep tasks the model currently passes between one time in five and four times in five. Below that it cannot learn yet. Above it, it already knows. Two more gates run at the same time. * **Duplicates.** The stand-in agent repeats itself, and so do real agents at low temperature. Identical tries carry no difference to learn from. * **What the reward is really tracking.** A model learns whatever gets the score. If longer replies happen to score higher, it learns to be long. The scan checks every reward against features like length and hedging, within each task, and warns when one predicts the reward. That warning is a reason to look at the judge before you train, not a reason to skip it. ## Run it ```python theme={"theme":"vitesse-dark"} import whileai as wai @wai.tool def get_order(order_id: str) -> dict: """Look up an order by id.""" ... data = wai.simulate( wai.seeded_agent([get_order]), tools=[get_order], system_prompt="Help customers with orders.", simulator=False, mode="rl", repeats=4, repeat_policy="fixed", budget=64, seed=0, ) scored = data.grade(judge=lambda row: {"reward": int(not row["seeded"])}) print(scored.select(mode="sft")) # the passes print(scored.select(mode="rl")) # the middle ``` ```text theme={"theme":"vitesse-dark"} sft selection: kept 16 of 64 rows eligible 16 (reward >= 1.0), junk 0, not passing 21 distinct behaviors: 6, covered: 6 at most 4 completion(s) per prompt; rejection-sampling selection wants 10 to 30 so the pick is not biased (rlhfbook.com/c/10-rejection-sampling.html; Llama 3 samples 10 to 30). Raise repeats= if you mean to choose among completions rather than filter. rl selection: kept 29 of 64 rows band 20%..80% pass rate: 0 asks dropped (0 too easy, 0 too hard) unanimous groups dropped: 5; duplicates dropped: 27; truncated drop: 0 groups kept: 11 hack scan: train warning: 27 duplicate rollout(s) within 13 ask(s) dropped warning: reward punishes reply length (corr -0.54); check the judge before training warning: reward punishes boilerplate (corr -0.31); check the judge before training warning: reward punishes hedging (corr -0.36); check the judge before training warning: 2 rollouts per ask at the median; the floor is coarse below 4, re-scan at repeats>=8 before acting on a close call warning: Difficulty was measured from 4 rollouts per task, so a task's band assignment can be off by about ±0.3. Use repeats=16 for a firmer band (the count the 20-80 band is measured from, rlhfbook.com/c/07-reasoning). warning: pass^k / pass@k do not survive the prune: the graded rows scored k=4, the selection leaves 2 rollout(s) per ask, so pass_at on these rows reports them as n/a. pass@1 and the carried calibration stamp still hold the graded measurement; take the k-way numbers from pass_at before optimize ``` The warnings are the point of the call, so read three of them. * **unanimous groups dropped: 5.** Five asks where all four tries scored the same. Nothing to learn there. * **reward punishes reply length (corr -0.54).** Here the judge reads the answer key, and the planted mistakes add words (an apology, a hedge), so shorter replies really are the better ones. On a real judge this line means: check whether it is grading the job or the word count. * **Use repeats=16 for a firmer band.** Four tries is a coarse estimate of a task's pass rate. The band is measured from sixteen in the source it cites. Sixty-four rows is a lesson, not a training set. ## Where it comes from 1. Shao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024. GRPO: the group average as the baseline. 2. Yu, Q. et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476, 2025. Dropping unanimous groups while sampling. 3. Yuan, Z. et al. Scaling Relationship on Learning Mathematical Reasoning with Large Language Models. arXiv:2308.01825, 2023. Keeping the passes for SFT, called rejection sampling. 4. Gao, L., Schulman, J., Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML, 2023. arXiv:2210.10760. Why a model learns what the score rewards rather than what you meant. 5. Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025\. Chapters *Rejection Sampling*, *Reasoning and Inference-Time Scaling* (the 20 to 80 band) and *Over-Optimization*. ## Next [Training is done when the held-out score moved](/learn/train-and-prove): export, train, and prove it.