> ## Documentation Index > Fetch the complete documentation index at: https://docs.withwhile.com/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > Install with `uv add whileai`; import as `import whileai as wai`. > Run the offline path first (`simulator=False`, `wai.seeded_agent`, a callable judge); no key is needed for it. > Report every pass rate with its interval and n, as `scored.pass_at` prints it. # GRPO, OPSD and OPD on prime-rl, on Modal, with your keys > Three ways to a per-token training signal on one taskset, from one config writer, on two GPUs each: the taskset's reward (GRPO), the same model shown the answer (on-policy self-distillation, OPSD), and a frozen stronger. The scripts are in the repository, not in the installed package. Clone it, then `cd recipes/04-train/prime-rl` before running the commands below. [Browse this recipe on GitHub](https://github.com/whilehq/whileai-sdk/tree/main/recipes/04-train/prime-rl). Three ways to a per-token training signal on one taskset, from one config writer, on two GPUs each: the taskset's reward (GRPO), the same model shown the answer (on-policy self-distillation, OPSD), and a frozen stronger checkpoint served next to the policy (on-policy distillation, OPD). Nothing here is a trainer. `wai.OPD` and `wai.OPSD` carry the cited defaults, `wai.prime_rl_config` writes the TOML prime-rl reads and says which knobs it honors, and `modal_prime_rl.py` runs Prime Intellect's published image on your Modal account. What you will learn: what distillation needs that RL does not (a teacher that beats the student, a shared tokenizer, a hint the model can use), what it does not need (a reward, a group with mixed outcomes), how the config writer reports a knob the trainer ignores, and how three arms on one held-out slice read side by side. You need a Modal account; the student and teacher are public 0.6B checkpoints and the taskset is bundled with prime-rl, so no other key. ## Run it ```bash theme={"theme":"vitesse-dark"} uv add whileai modal modal token set --token-id ... --token-secret ... cd recipes/04-train/prime-rl python run.py --validate # write the three configs, dry-run each on a CPU container python run.py # deploy, spawn the three runs (2 GPUs each), record call ids python run.py --collect # when they finish: results.json and the table ``` `configs/` holds the three TOMLs as written; every value in them is either a named default in `whileai/simulations/defaults.py` or an override named in `run.py`. `modal_prime_rl.py` takes any of them on its own: ```bash theme={"theme":"vitesse-dark"} modal run modal_prime_rl.py --config configs/opsd.toml --run-name opsd-1 modal run modal_prime_rl.py --config configs/opd.toml --run-name opd-1 --teacher PrimeIntellect/Qwen3-0.6B-Reverse-Text-RL ``` ## The three arms | Arm | Signal | What the teacher sees | Prompts a step | Samples a prompt | | ------ | ------------------------------------------------------------------------- | --------------------------------------- | -------------- | ---------------- | | `grpo` | reward: LCS ratio of the reply to the reversed text, minus the group mean | nothing; there is no teacher | 128 | 16 | | `opsd` | reverse KL, per token, to the same model | the task's `answer` in a system message | 32 | 1 | | `opd` | reverse KL, per token, to `Qwen3-0.6B-Reverse-Text-RL` | the same prompt, nothing else | 128 | 16 | Shared by every arm: student `PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT`, taskset `reverse-text` (reverse a sentence character by character), the last 128 of its 1,000 prompts held out (`dataset_split = "train[872:]"`, never trained on), 20 optimizer steps, learning rate 3e-6 on full weights, 128 completion tokens, the `prime-qwen3` renderer. The learning rate and the token cap are prime-rl's own `configs/debug/algo/{opd,self_distill}.toml` values; prime-rl evaluates those on the training rows, this recipe does not. The container is `ghcr.io/primeintellect-ai/prime-rl:v0.8.1.dev63`, one published tag per commit (this one is commit `e6d2b3f25`, 2026-08-21). The image ships its own `/usr/local/bin/python`; Modal injects its client's interpreter at the same path, so `prime-rl.Dockerfile` clears the symlink first and Modal's step runs after it. For OPD the launcher starts `uv run inference` for the teacher on GPU 0 at 0.4 of its memory, next to the policy's own engine at 0.4, the layout prime-rl's OPD configs use, and waits for `/v1/models` before `rl` starts. ## What the writer reports `wai.prime_rl_config` prints, per arm, which knobs prime-rl reads and where they landed, which it ignores and why, and the launch line: ```text theme={"theme":"vitesse-dark"} prime-rl config: configs/opd.toml method opd, model PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT, taskset reverse-text, 2 GPUs (1 inference, 1 trainer) 20 steps x 128 prompts x 16 rollouts, max_off_policy_steps 8 reads: teacher -> orchestrator.algo.teacher (name, base_url; key from VLLM_API_KEY); samples -> orchestrator.group_size; ... ignores: top_k=32: prime-rl scores the teacher's full-vocabulary prefill; the top-k support is not a knob there run: uv run rl @ configs/opd.toml ``` `wai.OPSD()` defaults to an EMA-anchored teacher (`anchor="ema:0.01"`, the SDFT and SDPO choice); prime-rl scores against the live policy, so the recipe passes `anchor="live"` and the config says what it runs. `wai.OPD` refuses a chat API as the teacher: the teacher is scored on the student's tokens, which needs a server that returns prompt log-probabilities. ## Result Run `e2e1`, 2026-09-20, three H100:2 containers, whileai 1.04, image `v0.8.1.dev63`. The number is the taskset's own reward, the LCS ratio of the reply to the true reversal, on the 128 held-out prompts, paired step 1 to step 20 by prompt, interval from `wai.compare` (paired bootstrap, 95%). The noise floor is the three step-1 scores of the same untrained student, one per arm: 0.082, 0.070 and 0.109, so `run_std` 0.020 and a delta under 0.121 is noise. | Arm | Held-out LCS, step 1 | step 20 | Delta \[95%] | Verdict | Replies cut at 128 tokens, step 1 to 20 | Wall clock | | ------ | -------------------- | ------- | ------------------------ | ------- | --------------------------------------- | ---------- | | `grpo` | 0.082 | 0.806 | +0.724 \[+0.678, +0.767] | moved | 90% to 0% | 884 s | | `opsd` | 0.070 | 0.252 | +0.182 \[+0.110, +0.254] | moved | 90% to 66% | 484 s | | `opd` | 0.109 | 0.834 | +0.725 \[+0.673, +0.772] | moved | 88% to 1% | 894 s | Head to head at step 20 on the same prompts: `opd` against `grpo` is +0.028 \[+0.013, +0.045], inside the 0.121 eval noise band, so the two are the same; `opsd` against `grpo` is -0.554 \[-0.614, -0.495]. Exact reversals (LCS 1.0) stayed at 0 of 128 in every arm, so the pass\@1 line of the report is flat by construction and the LCS marker is the result. The logged per-token reference KL, the number the distillation arms minimize: `opd` -0.294 at step 1 to -0.108 at step 20; `opsd` -0.076 to -0.073. Everything above is in `results.json`, per step, with each arm's config as run and `wai.compare`'s printed report. ## Learned * **A frozen teacher matched the reward in twenty steps, with no reward.** OPD and GRPO land on the same held-out number (the head-to-head is inside the noise band) from the same student, the same steps and the same learning rate. OPD paid for a second served model; GRPO paid for a verifier. On a task with a program for a reward, GRPO is the cheaper signal; on one without, OPD is the one that exists. * **The hint has to move the teacher for self-distillation to move the student.** OPSD's reference KL sat at -0.08 for the whole run against OPD's -0.29: showing this 0.6B model the answer barely changed what it predicted for its own tokens, which is the in-context-learning floor the SDFT and SDPO papers put at about 7B \[2, 3]. The writer's warning says so before the run; the run shows what it looks like. * **Watch the cap.** Every arm started with nine in ten replies cut at 128 tokens. The two arms that learned drove that to zero; the one that did not stayed at two in three, and its reward never left the truncated regime. Reply length is the first curve to read on any of the three. ## References 1. Agarwal, R. et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR 2024. arXiv:2306.13649. 2. Shenfeld, I. et al. Self-Distillation Enables Continual Learning. arXiv:2601.19897, 2026. 3. Hübotter, J. et al. Reinforcement Learning via Self-Distillation. ICML 2026. arXiv:2601.20802. 4. Zhao, S. et al. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. arXiv:2601.18734, 2026. 5. Kaur, S. et al. Rethinking On-Policy Self-Distillation for Thinking Models. arXiv:2607.05184, 2026. 6. Li, Y. et al. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. arXiv:2604.13016, 2026. 7. Shao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024. 8. Prime Intellect Team. INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learning. arXiv:2505.07291, 2025.