The scripts are in the repository, not in the installed package. Clone it,
then
cd recipes/04-train/prime-rl before running the commands below. Browse this recipe on GitHub.wai.OPD and wai.OPSD carry the cited defaults,
wai.prime_rl_config writes the TOML prime-rl reads and says which knobs
it honors, and modal_prime_rl.py runs Prime Intellect’s published image
on your Modal account.
What you will learn: what distillation needs that RL does not (a teacher
that beats the student, a shared tokenizer, a hint the model can use), what
it does not need (a reward, a group with mixed outcomes), how the config
writer reports a knob the trainer ignores, and how three arms on one
held-out slice read side by side. You need a Modal account; the student and
teacher are public 0.6B checkpoints and the taskset is bundled with
prime-rl, so no other key.
Run it
configs/ holds the three TOMLs as written; every value in them is either a
named default in whileai/simulations/defaults.py or an override named in
run.py. modal_prime_rl.py takes any of them on its own:
The three arms
Shared by every arm: student
PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT,
taskset reverse-text (reverse a sentence character by character), the
last 128 of its 1,000 prompts held out (dataset_split = "train[872:]",
never trained on), 20 optimizer steps, learning rate 3e-6 on full weights,
128 completion tokens, the prime-qwen3 renderer. The learning rate and
the token cap are prime-rl’s own configs/debug/algo/{opd,self_distill}.toml
values; prime-rl evaluates those on the training rows, this recipe does not.
The container is ghcr.io/primeintellect-ai/prime-rl:v0.8.1.dev63, one
published tag per commit (this one is commit e6d2b3f25, 2026-08-21). The
image ships its own /usr/local/bin/python; Modal injects its client’s
interpreter at the same path, so prime-rl.Dockerfile clears the symlink
first and Modal’s step runs after it. For OPD the launcher starts
uv run inference for the teacher on GPU 0 at 0.4 of its memory, next to
the policy’s own engine at 0.4, the layout prime-rl’s OPD configs use, and
waits for /v1/models before rl starts.
What the writer reports
wai.prime_rl_config prints, per arm, which knobs prime-rl reads and
where they landed, which it ignores and why, and the launch line:
wai.OPSD() defaults to an EMA-anchored teacher (anchor="ema:0.01", the
SDFT and SDPO choice); prime-rl scores against the live policy, so the
recipe passes anchor="live" and the config says what it runs. wai.OPD
refuses a chat API as the teacher: the teacher is scored on the student’s
tokens, which needs a server that returns prompt log-probabilities.
Result
Rune2e1, 2026-09-20, three H100:2 containers, whileai 1.04, image
v0.8.1.dev63. The number is the taskset’s own reward, the LCS ratio of
the reply to the true reversal, on the 128 held-out prompts, paired step 1
to step 20 by prompt, interval from wai.compare (paired bootstrap, 95%).
The noise floor is the three step-1 scores of the same untrained student,
one per arm: 0.082, 0.070 and 0.109, so run_std 0.020 and a delta under
0.121 is noise.
Head to head at step 20 on the same prompts:
opd against grpo is
+0.028 [+0.013, +0.045], inside the 0.121 eval noise band, so the two are
the same; opsd against grpo is -0.554 [-0.614, -0.495]. Exact
reversals (LCS 1.0) stayed at 0 of 128 in every arm, so the pass@1 line
of the report is flat by construction and the LCS marker is the result.
The logged per-token reference KL, the number the distillation arms
minimize: opd -0.294 at step 1 to -0.108 at step 20; opsd -0.076 to
-0.073. Everything above is in results.json, per step, with each
arm’s config as run and wai.compare’s printed report.
Learned
- A frozen teacher matched the reward in twenty steps, with no reward. OPD and GRPO land on the same held-out number (the head-to-head is inside the noise band) from the same student, the same steps and the same learning rate. OPD paid for a second served model; GRPO paid for a verifier. On a task with a program for a reward, GRPO is the cheaper signal; on one without, OPD is the one that exists.
- The hint has to move the teacher for self-distillation to move the student. OPSD’s reference KL sat at -0.08 for the whole run against OPD’s -0.29: showing this 0.6B model the answer barely changed what it predicted for its own tokens, which is the in-context-learning floor the SDFT and SDPO papers put at about 7B [2, 3]. The writer’s warning says so before the run; the run shows what it looks like.
- Watch the cap. Every arm started with nine in ten replies cut at 128 tokens. The two arms that learned drove that to zero; the one that did not stayed at two in three, and its reward never left the truncated regime. Reply length is the first curve to read on any of the three.
References
- Agarwal, R. et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR 2024. arXiv:2306.13649.
- Shenfeld, I. et al. Self-Distillation Enables Continual Learning. arXiv:2601.19897, 2026.
- Hübotter, J. et al. Reinforcement Learning via Self-Distillation. ICML 2026. arXiv:2601.20802.
- Zhao, S. et al. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. arXiv:2601.18734, 2026.
- Kaur, S. et al. Rethinking On-Policy Self-Distillation for Thinking Models. arXiv:2607.05184, 2026.
- Li, Y. et al. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. arXiv:2604.13016, 2026.
- Shao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024.
- Prime Intellect Team. INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learning. arXiv:2505.07291, 2025.