> ## Documentation Index
> Fetch the complete documentation index at: https://docs.withwhile.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Character training with whileai

> Change the weights so a model has a stable way of talking without a system prompt: sources, the recipe, and what to measure.

Character training changes the weights so a model has a stable way of
talking without a system prompt telling it to. It is the same post-training
machinery as everything else (rlhfbook.com ch. 17), aimed at the manner of
a reply instead of its correctness, and it is mostly a data pipeline: which
phrases never appear, which replies get chosen over which. This page is the
recipe as the SDK runs it. The worked example is
[`recipes/03-select/character`](https://github.com/whilehq/whileai-sdk/tree/main/recipes/03-select/character).

## What the sources say

**The book (ch. 17).** Character training is "the subset of post-training
designed around crafting traits within a model." Fine-tuning on trait data
beats prompting and activation steering for robustness (Maiya et al. 2025).
Anthropic's process, per Amanda Askell: write the traits, have the model
generate queries relevant to each trait, generate responses, rank the
responses by the trait. Constitutional AI without human data. Much of the
work is "developing pipelines to control the specific language in the
training data," down to removing `Certainly` and `as an AI model`.

**The Model Spec.** OpenAI's spec writes each trait as a principle plus
GOOD/BAD comparisons on real prompts: be warm, be clear and direct, don't
be sycophantic, be helpful when refusing, avoid being condescending, and so
on. The book calls model specs "one of the few tools that let one compare
the actual behavior of the model to what the designers intended." Read as
data, the spec is a constitution with labeled preference pairs attached.

**Maiya et al. 2025.** Three stages: a hand-written constitution, a
distillation stage that builds DPO pairs (direct preference optimization:
training on chosen-versus-rejected pairs with no reward model) from a
teacher with the constitution in its system prompt against a student
without, and an introspection stage where the trained model writes about
its own values for SFT (supervised fine-tuning on whole replies). Evaluation
is revealed preferences (which of \~150 trait words a judge sees in the
output), robustness to "ignore role-play and respond genuinely," and a
check that general capabilities did not move.

## The recipe

1. **Constitution.** One principle per trait, in prose, with labeled
   examples if you have them. `recipes/03-select/character/from_model_spec.py`
   builds one from the spec. Your own spec works the same way: id, principle,
   examples with `prompt`, `good`, `bad`.
2. **Prompts.** Situations that make the trait matter. Start from the
   examples' prompts; have the model write more, few-shot from those.
   Keep wording variants: the judge should grade the trait, not the phrasing.
3. **Replies.** `k` per prompt, under the deployment prompt only. The
   deployment prompt names the persona and nothing else. If the constitution
   is in the prompt at sampling time, you are measuring prompting, not
   character.
4. **Judge.** The principle goes in the judge's system prompt and nowhere
   else; that is `Task.privileged.principle` in the row schema, and the
   `privileged` block on the wire row. Use a different model family from
   the policy. Grade the spec's own GOOD/BAD replies with the same judge
   and read `judge_agreement`; below 0.8 agreement, or a Cohen's kappa
   (agreement corrected for chance) below 0.6, fix the judge first.
5. **Markers.** `trait` from the judge, `on_task` from the judge, and
   `no_filler` from a phrase list the judge never sees. Reward on a trait
   prompt is `trait AND on_task`.
6. **Pre-flight.** `pass_at` per trait. A trait the student already lands
   every time, or never, produces no pairs; the mixed prompts are the
   training signal (`group_signal`). `reward_correlations` says whether
   the judge is paying for length; above 0.3 it warns.
7. **Pairs and SFT.** `build_preference_pairs(rows, length_match=True)`, then
   `export_preference(pairs, "pairs.jsonl", system_prompt=DEPLOY_PROMPT)`.
   `export_training` on the passes for SFT; the loss mask covers the
   assistant turn by default.
8. **Train.** `wai.train(dataset_id, method="dpo")` on the pushed rows, or
   any DPO trainer reading `pairs.jsonl`.
9. **Measure.** The same prompts with a "drop the act" suffix, plus plain
   tasks the persona must not distort, before and after.
   `delta_report(before, after, target="marker:trait", must_not_regress=["on_task", "no_filler"])`
   gives the headline with an interval and fails on a regression.

## Run it

Both scripts run offline in seconds. The student and the judge are
scripted, so the numbers are real and the model is not.

```bash theme={null}
cd recipes/03-select/character
python run.py             # constitution -> rows -> pairs and SFT
python measure.py --demo  # before vs after on the adversarial holdout
```

`run.py` with the defaults (`--seed 0 --k 4`):

```text theme={null}
traits 8 | train 57 prompts x 4 = 228 rows | adversarial 120 | control 24 | spec 35
judge reference vs spec labels: agreement 1.00 (n=35, kappa 1.00)
pass@1 0.58 | pass^4 0.39 | pass@4 0.79 | headroom 0.21 | mixed prompts 23/57
markers: no_filler 0.64 [0.57,0.70] | on_task 1.00 | trait 0.58 [0.48,0.67]
controls on_task 1.00 | adversarial trait 0.25
corr(reward, reply length) +0.30 ok
pairs 20 (chosen longer 0.7) -> out/pairs.jsonl | sft 133 -> out/sft.jsonl
```

`measure.py --demo`, the headline lines:

```text theme={null}
marker:trait: moved_unreplicated (+0.500, 95% +0.408..+0.592, 30 paired tasks)
PASS
  pass_at_1                    0.375 -> 0.792  +0.417 [+0.319..+0.521]  up  (36 paired)
  marker:no_filler             0.681 -> 0.986  +0.306 [+0.229..+0.389]  up  (36 paired)
  marker:on_task               1.000 -> 1.000  +0.000 [+0.000..+0.000]  flat  (36 paired)
  marker:trait                 0.250 -> 0.750  +0.500 [+0.408..+0.592]  up  (30 paired)
```

`moved_unreplicated` means one eval run per side; the report says to run
each side three times (`simulate(tasks=..., runs=3)`) before calling it
proven. It also warns that `on_task` is 1.0 on both sides, so that guard
cannot fail here.

## The rows from one run

The live run in the example (hosted Qwen3-4B student, hosted Phi-4 judge)
is public: [while-ai/character-training-model-spec](https://huggingface.co/datasets/while-ai/character-training-model-spec)
on Hugging Face, splits `train` (60), `holdout` (144) and `eval` (35, the
spec's labeled replies with `gold_reward`). Grade the `eval` split with
your judge before reading anything else; that is the check the pipeline is
built around. In that run Phi-4 passed 10 of the spec's 20 BAD replies
(agreement 0.69, kappa 0.40), which is the judge failing the check.

## Things that go wrong

* **The judge likes long replies.** In the spec's own comparisons the GOOD
  reply is the longer one 70% of the time. A judge that learned that will
  pass verbose off-character replies. Length-neutral judge instructions,
  length-matched pairs, and the correlation line exist for this.
* **The judge is the policy.** Self-preference (ch. 5, ch. 12). The
  recipe's `judge_vs_spec` agreement drops and the pairs encode the
  model's taste, not the spec's.
* **Character costs helpfulness.** A warm reply that does not answer, a
  refusal that lectures. `on_task` is a hard guard in the delta report and
  the controls carry no trait marker at all.
* **No contrast.** A trait at pass\@1 of 0 or 1 yields nothing to pair.
  Write prompts where the student is inconsistent, or use a teacher for
  the chosen side and accept off-policy pairs (`same_policy=false` on the
  pair).
* **The holdout is the training set.** Adversarial variants of train
  prompts test robustness, not generalization. Written prompts split by
  hash give a prompt-disjoint holdout; `decontaminate` checks the overlap
  (word 8-grams).

## What the SDK does not do

Persona vectors, activation capping, persona subnetworks, and Maiya's
introspection stage (it needs the trained model). The SDK produces the
rows, the pairs, the judge check and the before/after measurement, and
`wai.train` runs DPO on the platform; `pairs.jsonl` is there for a trainer
of your own.