The scripts are in the repository, not in the installed package. Clone it,
then
cd recipes/04-train/dpo before running the commands below. Browse this recipe on GitHub.recipes/04-train/grpo: the refund assistant with one testable rule.
GRPO samples during training and learns from a reward; DPO learns offline
from chosen/rejected pairs and never samples while it trains. This example
is the second one, end to end: pairs, TRL’s DPOTrainer with a LoRA
adapter, the reward margin on the training page as it runs, and a
before/after on a holdout when it is done.
Run it
../grpo/reward.py, and
the same --seed builds the same set and the same holdout on every run
(119 prompts, 94 train, 25 holdout at --seed 0, on every Python from
3.10 to 3.14), so two DPO runs at one seed are paired on one holdout.
Where the pairs come from
On-policy (default). The base policy is sampled 8 times per train prompt. Every reply is scored by the rule in../grpo/reward.py, and
wai.build_preference_pairs pairs a higher-scoring reply with a lower one
from the same prompt, taking the rejected reply closest in length to the
chosen one. Both sides come from the policy being trained, which is where
DPO works best; length matching keeps the trainer from learning “longer is
better” before it learns the rule. Prompts the policy always passes or
always fails give no pair; the run reports how many prompts had contrast.
Exported. --pairs takes a file written by wai.export_preference
from any graded rows, for example a set pulled from the platform:
pairs.py turns either supply into TRL’s conversational shape: the prompt
is the system and user turns, chosen and rejected are one assistant turn
each. A rollout’s first turn is its first tool call rendered as a
<tool_call> block, or its first reply. A pair whose two first turns read
the same is dropped: the contrast was later in the rollout and a first-turn
trainer cannot learn it.
What lands on the dashboard
TrainerCallback forwards TRL’s DPO log: loss, reward_chosen,
reward_rejected, reward_margins (the implicit reward gap the loss is
pushing up), reward_accuracies (how often chosen beats rejected), and the
log-probabilities. pass@1 on the holdout before and after is the number
that matters; run.delta puts it on the run page with an interval, with
well_formed guarded so a run that learned the rule by breaking the wire
format is called out.
Knobs
A run
400 situations gave 72 unique prompts, 58 train and 14 holdout. The base policy passed 9% of holdout attempts; 4 samples per prompt found contrast on 8 prompts, 9 pairs, chosen the longer side in 5 of 9. Sixty steps on 9 pairs is thirty epochs, and the holdout still read pass@1 0.09 to 0.66, +0.57 [+0.38, +0.75],moved, with well_formed flat at 1.0. Few pairs is
the usual DPO problem, hence the default of 8 samples per prompt now, and
the pair_report in the summary says how many prompts had contrast.
On the model-written set (--prompts-file recipes/04-train/grpo/prompts.jsonl,
holdout 159 prompts): 308 pairs from 193 prompts with contrast, pass@1
0.17 [0.13, 0.22] to 0.69 [0.63, 0.75], +0.49 [+0.37, +0.60], moved.
See the GRPO README for the set and the side-by-side.
Round two
--from-run <run name> merges that run’s adapter into the base weights,
samples fresh pairs from the merged policy, and trains a new adapter with
the merged policy as the reference: iterated on-policy DPO. The merged
policy is saved under merged/ on the volume for serving or a third round.
moved. Two rounds of DPO
(0.17 to 0.92) passed 120 steps of GRPO (0.85) on this environment. The
round-two “before” (0.63) is the round-one policy resampled; it sits inside
round one’s after interval, which is what a stable measurement looks like.
The pairs go stale as the policy moves, which is why each round samples
its own; GRPO does the same every step.
Constructed negatives
Round two from the balanced adapter made the invented-id habit worse (no-id pass@1 0.82 to 0.26): DPO learns only from prompts with a pass and a fail, the base almost never invents an id on a no-id prompt, so those prompts never pair, and the with-id pairs teach “call the tool” across every kind of prompt.--constructed-negatives puts the contrast where
it is missing: for every no-id or off-topic prompt the policy answered
without a tool call, that reply is the chosen side and an invented
lookup_order call (an id derived from the prompt, never in it) is the
rejected side. On-policy on the chosen side, the one named mistake on
the other. pairs.constructed_negatives builds them; the pair report
counts them as constructed_pairs.
--balance 0.25 repeating the no-id
prompts, round one from the base built 411 constructed pairs against 351
sampled ones and learned “never call”: with_id 0.11 to 0.05, overall
DOWN. Round two from the balanced adapter built 404 against 442 and
repaired no_id (0.87 to 0.93, tool-call rate 0.07) while with_id slipped
0.60 to 0.54, flagged DOWN by the group table. So the constructed side
is one pair per distinct prompt (balance repeats do not multiply it) and
capped at constructed_share of the sampled pairs, 0.3 by default.
Capped, round one from the base (98 constructed against 377 sampled
pairs): overall 0.27 to 0.44, with_id 0.11 to 0.30, no_id 0.97 to 0.99
with no tool calls at all, off_topic 0.92 to 0.99. The first DPO run on
this set with no group moving the wrong way, and the slowest with-id gain
of any: the constructed side costs with-id learning at a fixed 60 steps.
The knob is constructed_share; the table says what each setting buys.
Capped, round two from the balanced adapter (95 constructed against 444
sampled): overall 0.62 to 0.77, with_id 0.56 to 0.75, no_id 0.73 to 0.69
with the tool-call rate at 0.28, off_topic 0.94 to 0.98. Side by side,
round two from the same adapter:
No setting wins both columns at this budget. The default holds the no-id
line where a plain second round breaks it; the uncapped setting repairs
no-id at with-id’s expense. What the group table adds is that the choice
is visible before the model ships, in the run’s own numbers, instead of
after.