On-policy distillation

​

What the research says

• The loss is a per-token reverse KL on the student’s own samples. GKD defines the family: sample from the student, score each token under a frozen teacher, minimize a divergence per token [1]. The reasoning line writes it as an RL update with advantage A_t = log p_teacher(x_t) - log p_student(x_t), no discount, no reference KL, no task reward [2]. prime-rl’s opd, TRL’s DistillationTrainer and the Tinker cookbook compute this.
• It is an order of magnitude cheaper than RL for the same number. Qwen3-8B reaches AIME’24 74.4 with OPD at 1,800 GPU hours against 67.6 with RL at 17,920 [3]. The signal lives on the teacher’s top-k support: OPD works by raising student/teacher top-k overlap, and k of 4 or more matches the sampled-token loss while k of 1 fails [7]; masking the signal to the teacher’s top 32 keeps it off filler tokens late in long replies [8].
• The failure modes are ceiling, tokenizer and initialization. The student saturates at the teacher [2]. A tokenizer mismatch silently drops the signal [9]. A student that cannot produce teacher tokens needs an off-policy SFT cold start first [7].
• Self-distillation beats GRPO where reward variance is zero. With the reference answer as the hint, Qwen3-8B reaches AIME’24 77.8 against GRPO’s 76.4 at one rollout per prompt and a fifth of the steps, because over half of GRPO’s batches had zero reward variance [4]. With a successful rollout and the environment’s feedback as the hint, LiveCodeBench v6 goes 41.2 to 48.8 with four times fewer generations [5]. With a demonstration, a model learns a new task while keeping the old ones (SDFT: 70.2 vs 66.2 new-task, 64.5 vs 53.4 retained) [6].
• The teacher must be anchored, and self-distillation hurts thinking models. Unregularized self-distillation diverges; SDFT and SDPO hold the teacher as a moving average of the student [5, 6]. On Qwen3-4B and 8B with thinking on, an answer-conditioned teacher costs 5.7 points avg@16 on AIME and HMMT, because the signal suppresses the high-entropy fork tokens where the model checks itself [10]; the hint cuts “wait” and “maybe” tokens by about 95% [11]. The fix is to route: failed rollouts self-distill, correct ones take the reward [12].

​

The calls

import whileai as wai

teacher = wai.Endpoint(
    url="http://localhost:8001/v1", model="PrimeIntellect/Qwen3-0.6B-Reverse-Text-RL"
)

opd = wai.OPD(teacher)  # reverse KL, top_k 32, 4 samples, T 1.0, 8k cap
opsd = wai.OPSD(privileged="answer")  # the same model shown the task's answer, 1 sample
cfg = wai.prime_rl_config(
    "reverse-text",
    opsd,
    model="PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT",
    out="opsd.toml",
)
print(cfg)

prime-rl config: opsd.toml
  method opsd, model PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT, taskset reverse-text, 2 GPUs (1 inference, 1 trainer)
  100 steps x 64 prompts x 1 rollouts, max_off_policy_steps 8
  reads: privileged -> orchestrator.algo.demo_key = 'answer' (read from the task's info, then its top-level fields); template -> orchestrator.algo.template; samples -> orchestrator.group_size; temperature -> orchestrator.train.sampling.temperature; max_tokens -> orchestrator.train.sampling.max_completion_tokens; learning_rate -> trainer.optim.lr = 5e-06
  ignores: anchor=ema:0.01: prime-rl opsd scores against the live policy; an EMA or initial-weights teacher (SDFT, SDPO) is not offered there
  warning: OPSD costs points on thinking models (Kaur et al. 2026, arXiv:2607.05184) and needs in-context learning strong enough to use the hint (about 7B up); run a GRPO arm on the same holdout before believing a number.
  run: uv run rl @ opsd.toml

​

Run it: three arms on one taskset

uv add whileai modal
modal token set --token-id ... --token-secret ...
cd recipes/04-train/prime-rl
python run.py --validate     # write the three configs, dry-run each on a CPU container
python run.py                # deploy, spawn the three runs (2 GPUs each), record the call ids
python run.py --collect      # when they finish: results.json with the paired deltas

​

Read the number

​

Next

​

References

1. Agarwal, R. et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR 2024. arXiv:2306.13649.
2. Lu, K. et al. On-Policy Distillation. Thinking Machines Lab, 2025; code in tinker-cookbook distillation/.
3. Yang, A. et al. Qwen3 Technical Report. arXiv:2505.09388, 2025.
4. Zhao, S. et al. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. arXiv:2601.18734, 2026.
5. Hübotter, J. et al. Reinforcement Learning via Self-Distillation. ICML 2026. arXiv:2601.20802.
6. Shenfeld, I. et al. Self-Distillation Enables Continual Learning. arXiv:2601.19897, 2026.
7. Li, Y. et al. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. arXiv:2604.13016, 2026.
8. Fu, Y. et al. Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes. arXiv:2603.25562, 2026.
9. Sun, J. et al. SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation. arXiv:2605.07711, 2026.
10. Kaur, S. et al. Rethinking On-Policy Self-Distillation for Thinking Models. arXiv:2607.05184, 2026.
11. Kim, J. et al. Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs? arXiv:2603.24472, 2026.
12. Li, G. et al. Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing. arXiv:2604.02288, 2026.
13. Penaloza, E. et al. Privileged Information Distillation for Language Models. arXiv:2602.04942, 2026.