> ## Documentation Index > Fetch the complete documentation index at: https://docs.withwhile.com/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > Install with `uv add whileai`; import as `import whileai as wai`. > Run the offline path first (`simulator=False`, `wai.seeded_agent`, a callable judge); no key is needed for it. > Report every pass rate with its interval and n, as `scored.pass_at` prints it. # DPO on Modal, with the dashboard watching > Direct preference optimization on the same environment as recipes/04-train/grpo: the refund assistant with one testable rule. The scripts are in the repository, not in the installed package. Clone it, then `cd recipes/04-train/dpo` before running the commands below. [Browse this recipe on GitHub](https://github.com/whilehq/whileai-sdk/tree/main/recipes/04-train/dpo). Direct preference optimization on the same environment as [`recipes/04-train/grpo`](/recipes/04-train/grpo): the refund assistant with one testable rule. GRPO samples during training and learns from a reward; DPO learns offline from chosen/rejected pairs and never samples while it trains. This example is the second one, end to end: pairs, TRL's `DPOTrainer` with a LoRA adapter, the reward margin on the training page as it runs, and a before/after on a holdout when it is done. ## Run it ```bash theme={"theme":"vitesse-dark"} uv add whileai modal modal profile activate export WHILEAI_API_KEY=... # for the dashboard; optional modal run recipes/04-train/dpo/train_modal.py modal run recipes/04-train/dpo/train_modal.py --pairs pairs.jsonl # your own pairs modal run recipes/04-train/dpo/train_modal.py --loss-type ipo --beta 0.1 # another DPO loss ``` Default: 200 prompts, 20% held out by scenario, Qwen2.5-1.5B-Instruct, 60 steps at 8 pairs a step, one A10G, about ten minutes. The run's URL is printed at the start. No key means the same run with the numbers printed at the end only. The prompts and the split come from `../grpo/reward.py`, and the same `--seed` builds the same set and the same holdout on every run (119 prompts, 94 train, 25 holdout at `--seed 0`, on every Python from 3.10 to 3.14), so two DPO runs at one seed are paired on one holdout. ## Where the pairs come from **On-policy (default).** The base policy is sampled 8 times per train prompt. Every reply is scored by the rule in `../grpo/reward.py`, and `wai.build_preference_pairs` pairs a higher-scoring reply with a lower one from the same prompt, taking the rejected reply closest in length to the chosen one. Both sides come from the policy being trained, which is where DPO works best; length matching keeps the trainer from learning "longer is better" before it learns the rule. Prompts the policy always passes or always fails give no pair; the run reports how many prompts had contrast. **Exported.** `--pairs` takes a file written by `wai.export_preference` from any graded rows, for example a set pulled from the platform: ```python theme={"theme":"vitesse-dark"} import whileai.simulations as wai rows = wai.pull("ds_...") pairs, report = wai.build_preference_pairs(rows) # a pass vs a fail per prompt wai.export_preference(pairs, "pairs.jsonl") ``` `pairs.py` turns either supply into TRL's conversational shape: the prompt is the system and user turns, chosen and rejected are one assistant turn each. A rollout's first turn is its first tool call rendered as a `` block, or its first reply. A pair whose two first turns read the same is dropped: the contrast was later in the rollout and a first-turn trainer cannot learn it. ## What lands on the dashboard `TrainerCallback` forwards TRL's DPO log: `loss`, `reward_chosen`, `reward_rejected`, `reward_margins` (the implicit reward gap the loss is pushing up), `reward_accuracies` (how often chosen beats rejected), and the log-probabilities. pass\@1 on the holdout before and after is the number that matters; `run.delta` puts it on the run page with an interval, with `well_formed` guarded so a run that learned the rule by breaking the wire format is called out. ## Knobs | flag | default | what it does | | ----------------- | ------- | ---------------------------------------------------------------------------------------- | | `--steps` | 60 | optimizer steps (2 pairs a device, 4 accumulated) | | `--beta` | 0.1 | how far from the reference the policy may move | | `--loss-type` | sigmoid | any `DPOConfig.loss_type`: `ipo`, `hinge`, `robust`, ... | | `--pair-samples` | 8 | replies per prompt when building on-policy pairs; a policy that rarely passes needs more | | `--learning-rate` | 5e-6 | LoRA learning rate | | `--pairs` | | an `export_preference` JSONL instead of on-policy sampling | ## A run 400 situations gave 72 unique prompts, 58 train and 14 holdout. The base policy passed 9% of holdout attempts; 4 samples per prompt found contrast on 8 prompts, 9 pairs, chosen the longer side in 5 of 9. Sixty steps on 9 pairs is thirty epochs, and the holdout still read pass\@1 0.09 to 0.66, +0.57 \[+0.38, +0.75], `moved`, with `well_formed` flat at 1.0. Few pairs is the usual DPO problem, hence the default of 8 samples per prompt now, and the `pair_report` in the summary says how many prompts had contrast. On the model-written set (`--prompts-file recipes/04-train/grpo/prompts.jsonl`, holdout 159 prompts): 308 pairs from 193 prompts with contrast, pass\@1 0.17 \[0.13, 0.22] to 0.69 \[0.63, 0.75], +0.49 \[+0.37, +0.60], `moved`. See the GRPO README for the set and the side-by-side. ## Round two `--from-run ` merges that run's adapter into the base weights, samples fresh pairs from the merged policy, and trains a new adapter with the merged policy as the reference: iterated on-policy DPO. The merged policy is saved under `merged/` on the volume for serving or a third round. ```bash theme={"theme":"vitesse-dark"} uv run --with modal modal run recipes/04-train/dpo/train_modal.py --prompts-file recipes/04-train/grpo/prompts.jsonl --run-name refund-dpo-r1 uv run --with modal modal run recipes/04-train/dpo/train_modal.py --prompts-file recipes/04-train/grpo/prompts.jsonl --from-run refund-dpo-r1 --run-name refund-dpo-r2 ``` On the model-written set, round two from the 0.69 policy: 412 pairs from 261 prompts with contrast (the policy now fails less often, so fewer prompts split, but more pairs per split prompt), pass\@1 0.63 \[0.57, 0.69] to 0.92 \[0.90, 0.95], +0.26 \[+0.17, +0.35], `moved`. Two rounds of DPO (0.17 to 0.92) passed 120 steps of GRPO (0.85) on this environment. The round-two "before" (0.63) is the round-one policy resampled; it sits inside round one's after interval, which is what a stable measurement looks like. The pairs go stale as the policy moves, which is why each round samples its own; GRPO does the same every step. ## Constructed negatives Round two from the balanced adapter made the invented-id habit worse (no-id pass\@1 0.82 to 0.26): DPO learns only from prompts with a pass and a fail, the base almost never invents an id on a no-id prompt, so those prompts never pair, and the with-id pairs teach "call the tool" across every kind of prompt. `--constructed-negatives` puts the contrast where it is missing: for every no-id or off-topic prompt the policy answered without a tool call, that reply is the chosen side and an invented `lookup_order` call (an id derived from the prompt, never in it) is the rejected side. On-policy on the chosen side, the one named mistake on the other. `pairs.constructed_negatives` builds them; the pair report counts them as `constructed_pairs`. ```bash theme={"theme":"vitesse-dark"} uv run --with modal modal run recipes/04-train/dpo/train_modal.py --prompts-file recipes/04-train/grpo/prompts.jsonl --balance 0.25 --constructed-negatives ``` Uncapped, they overcorrect. With `--balance 0.25` repeating the no-id prompts, round one from the base built 411 constructed pairs against 351 sampled ones and learned "never call": with\_id 0.11 to 0.05, overall `DOWN`. Round two from the balanced adapter built 404 against 442 and repaired no\_id (0.87 to 0.93, tool-call rate 0.07) while with\_id slipped 0.60 to 0.54, flagged `DOWN` by the group table. So the constructed side is one pair per distinct prompt (balance repeats do not multiply it) and capped at `constructed_share` of the sampled pairs, 0.3 by default. Capped, round one from the base (98 constructed against 377 sampled pairs): overall 0.27 to 0.44, with\_id 0.11 to 0.30, no\_id 0.97 to 0.99 with no tool calls at all, off\_topic 0.92 to 0.99. The first DPO run on this set with no group moving the wrong way, and the slowest with-id gain of any: the constructed side costs with-id learning at a fixed 60 steps. The knob is `constructed_share`; the table says what each setting buys. Capped, round two from the balanced adapter (95 constructed against 444 sampled): overall 0.62 to 0.77, with\_id 0.56 to 0.75, no\_id 0.73 to 0.69 with the tool-call rate at 0.28, off\_topic 0.94 to 0.98. Side by side, round two from the same adapter: | constructed share | with\_id | no\_id | tool calls on no\_id | | ----------------- | ------------ | ------------ | -------------------- | | none | 0.56 to 0.91 | 0.82 to 0.26 | 0.63 | | 0.3 (default) | 0.56 to 0.75 | 0.73 to 0.69 | 0.28 | | uncapped | 0.60 to 0.54 | 0.87 to 0.93 | 0.07 | No setting wins both columns at this budget. The default holds the no-id line where a plain second round breaks it; the uncapped setting repairs no-id at with-id's expense. What the group table adds is that the choice is visible before the model ships, in the run's own numbers, instead of after.