whileai login saved, or WHILEAI_API_KEY, or api_key=. The pure-Python checks that need no key (optimize, hack_scan, judge_trust, delta_report and the rest) are here too, because they sit between a run and a push.
Sign in
~/.whileai/credentials.json and every platform call reads it from there. Interrupted before you approved? Run it again; it resumes the same code. This is the path for coding agents too: tell yours to run whileai login and click the link it shows you. whileai status shows which key is in use, whileai logout removes it.
No account yet, or no browser? One command creates the account and the key. Open the dashboard later by signing in with an email code.
whileai status prints the link. whileai status shows the tier; whileai.account() returns tier, limits and usage.
Store datasets on While
Push a run to your account so the rest of the loop can read it. Credentials resolve in this order:api_key= argument, WHILEAI_DELEGATED_CREDENTIAL (a short-lived zp_dc_... issued from a Clerk session), WHILEAI_API_KEY, then the key saved by whileai login.
whileai.account() says how much you have. parent= records dataset lineage so iterations show as a family on the platform.
The publish gate
data.push and wai.push_file run a publish gate first (gate=False skips it; wai.push_rows does not gate unless you pass gate=True, because the caller may already have run optimize). Every graded row gets a calibration stamp: its task’s pass rate over k repeats, k, and the policy that produced it, so a trainer can build a curriculum or retire solved tasks. An RL-shaped run (repeats of one ask) is refused with PublishGateError when it is ungraded or has no mixed group, because a grouped update would learn nothing from it. The report comes back as entry["gate"], with warnings when unanimous asks, or asks outside the difficulty band, are still present; wai.optimize(data, mode="rl") prunes those. wai.publish_gate(rows) runs the same check on any row list.
The stamp is the schema’s Calibration object: wai.calibration_of(row) reads it back typed, from_row carries it on rollout.extra["calibration"], and to_row writes it out again. k is the repeats the grader saw, not the rows that survived: optimize(mode="rl") stamps its selection from the rows it was given, before its own dedupe and trims, and the gate keeps a carried stamp rather than re-measuring it on what is left. The gate’s own pass_at block is still over the rows in front of it, and says so when the two differ.
Export training rows
export_dataset and export_training are the same function object (export_dataset is export_training): same arguments, same file, same report. Write export_dataset in new code; export_training is the older spelling, kept so nothing already written breaks. training_rows is the list-returning half of the same path, without writing a file.
Training rows carry a loss_mask, one 0/1 per message: 1 on the agent’s turns, 0 on system, user, and tool-output turns. Tool output is the environment’s text, not the policy’s, so a trainer should not learn to predict it. mask_mode="final" trains only the last assistant turn, for conversations whose earlier agent turns were scripted or came from another policy; the export report counts trained_messages and masked_messages. unroll=True turns an N-turn conversation into N samples, the k-th ending at the k-th agent turn with loss on that turn only, so every earlier turn trains once with the context it actually had (rlhf-book ch. 4). max_tool_output_chars= caps each tool message, appends a [... N chars of tool output truncated] marker and counts the cut on the row and in the report, so context spent on tool output is a decision the export makes out loud (ch. 13).
Two wire shapes come out of the exporters, and a trainer needs the second one:
format="openai" (the default) is the API wire row: the whole conversation in messages, function.arguments as a JSON string, and the ask alongside as prompt. format="trl" is what trl.data_utils.maybe_apply_chat_template accepts. For SFT that is conversational messages with no prompt string column: TRL decides “is this conversational?” from the column set, and a prompt string next to messages makes it skip the chat template silently and train on the bare ask; the ask survives as prompt_text. For preference data it is prompt as the message list up to the first agent turn with chosen/rejected as the completions only, because the default shape (a prompt string with full conversations on both sides) raises TypeError: string indices must be integers inside TRL. In the TRL shape function.arguments is a dict, not a JSON string: HF chat templates render it with | tojson, so a pre-encoded string is quoted twice and the student learns to emit a string where an object belongs. The tool_call_roundtrip block in the report names which of the two encodings it checked (encoding: "json_string" or "dict"), so invalid: 0 says what it actually vouches for.
The export refuses rows whose reply quotes their own privileged context (privileged_leak in the error) because the export scrubs the key, not the reply; drop the rows leak_report names, or pass validate=False.
Prune before training
optimize(mode="rl") drops, in this order: junk rows; duplicate rollouts within an ask (the same trajectory twice adds nothing to a group-relative advantage); truncated rollouts (truncated="keep" leaves them in as overlong, "penalize" keeps them as failures with the judged score under reward_before_penalty, DAPO’s overlong handling); unanimous asks (all pass or all fail: zero advantage); and asks outside the difficulty band. “Out of band” means outside the 0.2 to 0.8 pass-rate band, never off-topic. The filter does not read the prompt at all, so an on-topic ask the policy always solves is dropped and an odd one it solves half the time is kept. It then keeps whole groups round-robin across fault kinds and, within a fault kind, round-robin across pass rates: a 25% ask, a 50% ask and a 75% ask are taken in turn, with no preference for the middle (order="middle" restores the older nearest-to-50% ranking).
Each kept row’s calibration stamp carries pass_rate_ci95, the interval on that pass rate, and the report says so when the band was measured from fewer than 16 rollouts per task, since at 8 a task’s band assignment can be off by about 0.3. The prune shrinks every group, so the k-way reliability numbers do not survive it: pass_at on the selection reports pass^k and pass@k as n/a where the graded rows had them, which is why you print pass_at before this call. The report says so in hygiene_warnings, and the carried calibration stamp keeps the graded per-task measurement.
optimize(mode="sft") is rejection sampling (rlhf-book ch. 9): select="top_per_prompt" keeps each prompt’s highest-reward completion above min_reward (default 1.0; lower it for a partial-credit grader), "top_k_overall" the best k across prompts, and the random_* rules are the matching chance controls. Exported groups carry n0/n1 (fail/pass, partial credit splits at 0.5) and reward_mean/reward_std. The band is the offline difficulty filter from the reasoning-model recipes (keep prompts the policy solves 20-80% of the time); it is a heuristic, so it is a parameter.
Every selector report (select_for_rl, select_for_sft, build_preference_pairs) carries eval_sourced, the rows or pairs whose reward came from evaluate() (lineage.source == "eval"), with a warning when it is non-zero: a held-out score that becomes the reward makes the scorer you report the one you optimised against. Nothing is dropped; grade the training set with run_judge or data.grade and keep evaluate for held-out rows.
What will the policy learn?
hack_scan asks the question the same way: reward and every candidate feature are centered within ask, ranked by that correlation, and compared to a noise floor from shuffling reward within ask (tau). Features come in two tiers, both pure Python: the hand tier (reply length, tool calls, turns, truncation, surface counts, one indicator per tool called, mean token logprob, every numeric marker, plus features={"name": fn} of your own) and the auto tier (the 200 most common words and word pairs in the agent’s text, and pairwise ANDs that beat both parents), which is the tier that finds the shortcut nobody listed. endorsed names what the reward should track, as substrings of feature names; with it the scan can say reward_hack (the top feature is not endorsed, and the warning names what the policy would learn instead), integrity (share of the above-floor signal that is endorsed), and lists rivals. Without it the scan still ranks and floors. An agent that emits only a couple of distinct trajectories per ask makes every feature that separates them an exact function of the label. They all tie at |rho| 1, and the floor cannot break a tie between two perfect explanations, so the scan returns degenerate with top_feature None, lists the tied features in collinear, and names the cause (distinct_per_ask) rather than picking the alphabetical winner.
The whole loop, before, during and after training, is in Reward hacking and runs offline in recipes/02-measure/reward-hacking. optimize(mode="rl", endorsed=[...]) carries the scan as report["hack_scan"], with its warnings in report["hygiene_warnings"] next to the older pooled report["correlations"] (reply length, tool calls, turns, flagged at HACK_THRESHOLD 0.3). A reward that tracks a shortcut is a judge problem, so it is flagged, not pruned. The publish gate reports the same on RL-shaped rows, plus near-duplicate asks and length spread; data.push(endorsed=[...], strict_hacks=True) refuses a reward_hack. Standalone: wai.reward_correlations(rows), wai.dedupe_groups(rows), wai.near_duplicate_prompts(rows), wai.length_report(rows).
Curriculum: easy to hard, and retire the solved
A curriculum needs per-prompt difficulty (rlhf-book ch. 7), which is each task’s pass rate over its k rollouts.curriculum(rows) splits graded tasks into trainable (ordered easy to hard, and bucketed into tiers for a staged schedule), retired (pass rate above solved, default 0.8: an all-pass task is dead gradient), and not ready (below floor, default 0.2: no signal until the policy improves), and counts how many trainable tasks sit in the 20-80% band. The two defaults are the band’s own edges, so curriculum and optimize(mode="rl") agree on which tasks are trainable.
Agents
An agent exists the moment a push names it or a trace arrives withgen_ai.agent.name. Everything on the platform hangs off it.
From the terminal
The platform verbs a coding agent needs, as commands:--json and --api-key; errors exit 1 with the reason on stderr.
Thin calls into whileai.platform. The old whileai purge (ZeroProof traces
and datasets) is gone; wai.purge_agent("demo-agent") and
wai.delete_empty_datasets(max_rows=2) remain in Python, both with
dry_run=True, until the ingest module is retired.
Train, holdout, eval
scenario_id, so a task is wholly on one side, and the same task lands on the same side every run. A purpose="holdout" push warns when the set is too small to prove a 5-point gain at 80% power.
A task’s identity is its cell in the coverage grid: the tools, the situation axes, and at most one clause of the policy. Each clause owns its own block of cells and the cells that pair the other axes carry no clause, so editing the system prompt keeps every task except the ones for the clause that changed. Rewording one rule, adding one, or swapping the model leaves the rest of the eval paired for compare_runs.
Training data out of traces
The platform’s “Make training data” button, as one line:send_score(trace_id, value) grades a run that already ran. 1.0 or above is a pass, so a 0-to-1 quality number never reads as one; send that under its own name= and keep score for the verdict. Re-sending the same name is a correction. Emitting whileai.reward on the span does the same thing when your grader runs inline.
Runs of the same prompt are grouped by zeroproof.scenario_id. kind="rl" keeps the prompts the agent passes some of the time and not always (20% to 80% by default); kind="sft" keeps the best run of every prompt that ever passed. Either way the prompts are split into a train set and a held-out set. since="7d" narrows the window, band= and holdout= move the defaults, and any other keyword is a trace filter (model=, tool=, evalSet=).
Trust the numbers
Checks that decide whether a result is believable. All are report-only and run offline over rows you already have.{"key": ..., "label": 0 or 1} or a {key: label} dict, where a label names its row by key, by scenario_id plus rollout_index, or by prompt), attach them with attach_labels(rows, labels, kind="human"), and grade. That stamps gold_reward and gold_kind="human"; a model’s labels, or a second judge pass, are marked model and do not count, and older rows with gold_reward but no record of who wrote it count as unknown. Every grade call then ends by checking the judge against those labels, with no flag needed: the summary lands on every graded row as judge_meta["trust"] (agreement, agreement_low, kappa, n_gold, ok), in the grade report as trust, and in publish_gate as judge_trust. The judge passes when the lower bound of its agreement with the people is at least 0.80 and kappa at least 0.60; under either, the report says the number, the floor, and what to do. With no human labels the grade prints one line saying the judge was not measured. grade(trust="require") raises instead of printing; trust="off" skips the check. audit_grades never audits with the grader’s own model: when the auditor would be the same, it uses the other hosted model and the report says which (grader, auditor), or it stops and asks for backend_spec=.
judge_trust(rows, judge=) is the standalone version. report["ok"] means measured and clean, so with no labels it is False and the report says the judge is unmeasured rather than untrustworthy (format_judge_trust prints NOT MEASURED). The report gives agreement with a Wilson interval and Cohen’s kappa, agreement on two task halves (tune the rubric on one, read the other), judge pass rate on short versus long replies within the same human label (length bias the humans rule out), and, with the judge callable, a re-judge of a sample as-is (consistency) and with neutral filler appended (a flip means the judge reads length). Disagreements come back as a review queue. It refuses model gold unless allow_model_gold=True. The gold set needs both passes and failures; with one class only the report says so and skips the kappa and length flags. probes="all" (or a list) tries the reward hacks a policy finds first on the judge on purpose: filler, the rubric’s own words stuffed in, a claim of success with no evidence, the ask echoed back, a well-formed tool call with empty arguments, a sycophantic opener, a polite refusal. An additive probe is exploitable when failing replies start passing; a replacement probe when a reply with no content passes. report["exploitable_by"] names the holes at or over 10%, and a policy trained on this judge will find those same holes. Standalone: wai.judge_probes(rows, judge, rubric=...). With the hosted judge, call wai.grade once first (or whileai.simulations.score.grade_llm.warm_judge; it is not re-exported) so the cold start, two to three minutes, is not counted as timeouts.
Decontamination. Four rules between a dataset’s rows and any evaluation source (row lists, JSONL paths, or platform dataset ids), each counted on its own and a row counted once. A row is contaminated when it shares a scenario_id or task_id with an eval row (n_same_task: a task is a situation, not a string, so a rephrasing of an eval situation is the eval situation), when it is an eval prompt verbatim (n_exact), or when one eval text covers at least 80% of its words in shared 8-grams (n_near; overlap=, the Llama 2 rule; one shared 8-gram is not enough, because situations written from the same templates share whole sentences without sharing the question, and short prompts match verbatim only). fields=("prompt", "final_text") also checks replies against eval answers and references. Word overlap does not see a paraphrase: a holdout written by re-running the generator was 70% within 0.85 cosine of the training batch, and the 8-gram rule flagged 4 of its 101 prompts where a semantic pass flagged 16. Pass embedder= (any callable from a list of texts to one vector per text, so nothing is imported; with sentence-transformers, embedder=lambda texts: model.encode(texts, normalize_embeddings=True).tolist()) and rows whose prompt is within similarity= (0.85 cosine) of an eval prompt are flagged as n_semantic. That flag means the two prompts read alike, not that they are the same task: “cancel one reservation” and “cancel three reservations” for different customers score 0.93 with no shared answer. So the task-id rule decides first, the semantic pass only looks across different task ids, and report["notes"] says the flag is a question to check. The default stays lexical; the threshold was read off BGE (unrelated prompts score about 0.55 there) and needs picking for another model, so when the eval rows carry task ids the pass measures how alike distinct tasks read to your embedder (the 99th percentile of similarity over eval-prompt pairs with different task ids) and notes says it, and says when similarity= sits below it, since a threshold there flags tasks that merely share a domain. The report returns the clean rows with the first offenders, their coverage or similarity, and hits per field.
Intervals and comparison. Every pass@1 carries a 95% interval from a bootstrap over tasks (pass_at(rows).ci95), and metric_summary / marker_summary do the same for markers. pass^k and pass@k carry their own (pass_pow_k_ci95, pass_at_k_ci95), a bootstrap over the k-eligible groups. Markers come from the judge: return {"reward": ..., "markers": {"name": value}} from a grader= or run_judge callable and they land on row["markers"], which is what marker_summary, delta_report and from_row read. compare_runs pairs the tasks two runs share, bootstraps the paired difference, and adds a sign-flip permutation p-value; fewer than five shared tasks falls back to an unpaired test and says so. Tasks on one side only are dropped from a paired comparison; note says how many and paired_share is the fraction that paired, so a verdict over a quarter of the eval reads as one. Under half paired is situations in delta_report’s not_comparable: the two arms drew different situation sets (a hard_share, dimensions or seed change between them), the delta over the few that pair is between two evals, and the warning says to pin the after side to the baseline’s tasks (tasks=) or compare per tier with dataset_report. The verdict no_difference_detected means the interval covers zero, not that the runs are equal. A task is a situation, not a string: every report (pass_at, compare_runs, delta_report, eval_variance, curriculum, group_signal, the exporters) groups rows by wai.task_key(row), the engine’s scenario_id when the row has one, so repeats and rephrasings of one situation count as one task and the same rows give the same task count everywhere. pass_at(rows).config and delta_report(...)["config"] say what the rows were produced with (temperature, reply budget, policy and judge versions), and delta_report warns when the two sides differ.
Same tasks, new prompt. A run draws its tasks from the grid by seed and, above concurrency: 1, by completion order, so a second simulate() shares only part of its tasks with the first. To A/B a prompt edit, a model swap or another seed on exactly the same eval, pin the task set: wai.simulate(agent, tools=TOOLS, system_prompt=EDITED, tasks=base) re-runs every prompt of base (a run, its rows, or its JSONL path) on its own scenario_id, under the same faults and world state, and draws nothing new; it stops with tasks_done once every prompt has its rollouts, and compare_runs(base.rows(), rerun.rows()) pairs every task.
tasks= copies the prompts and, unless you pass repeats=, the pinned run’s k (the most rollouts any of its prompts has), so a base built with mode="rl", repeats=4 and re-run as simulate(..., tasks=base, mode="rl") comes back at k=4 and pass_at reports the same k on both sides. Pass repeats= to re-run at a different k on purpose:
holdout_size(effect, base=, k=) says how many paired tasks prove a gain at 80% power, and detectable_effect(n_tasks, ...) is the same solved for the gain. Its binomial model assumes the gain is spread evenly across tasks and the two arms are independent draws, and says so in notes; when a trait is only exercised by some prompts most tasks are ties, the paired differences spread far wider, and the model under-sizes by several times (a voice lane at 0 to 0.127 needed 54 tasks where the model said 14), so the model path also returns n_tasks_concentrated. On a holdout whose tasks differ in difficulty the model errs the other way, asking for 1 / (1 - Var(p_i) / (p(1-p))) times the tasks pairing needs (1.19x at spread 0.2 around 0.5, 2.78x at 0.4); before= alone reports the spread and that ratio. The honest paths measure: before=before, after=after (the same two row lists delta_report takes) reads the per-task paired sd off both arms of a previous eval on the same tasks, with the covariance pairing buys in it, and task_std= (the per-task sibling of run_std) takes the number you read off a delta_report ((hi - lo) * sqrt(n_paired_tasks) / 3.92 from target_ci95).
Before and after. delta_report runs compare_runs on pass@1 and every marker both row sets share. target= names the metric the training was meant to move and gives the headline; must_not_regress= names the behaviors whose significant drop fails the report; any other significant drop is a warning. format_delta_report(report) prints one line per metric. eval_variance(run_1, run_2, run_3) is the eval’s own re-run standard deviation (three or more evaluations of the same model); passing it as run_std= makes any delta inside the re-run band within_noise, and a target there reads within_eval_noise rather than moved, since re-running the eval moves it that much on its own (rlhf-book ch. 16). Pass run_std_runs= with it (the n_runs the floor came from) so the band uses the t quantile at runs - 1 degrees of freedom: a floor from three re-runs is an estimate, and the 1.96 band that reads it as exact lets about one pure-noise delta in five through. A bare run_std= keeps 1.96 and warns. by= names a row key, a marker, or a callable that groups rows (a prompt category, a tool, a persona); the report then carries groups, the target compared within each group, and groups_down for any group whose target dropped significantly while the headline moved. A headline over one dominant kind of prompt cannot hide the other kinds that way. Both sides should have the same rollouts per task; when a run lost some (data.report()["rollouts_lost"], with the reasons in rollouts_lost_by) and one arm sits at k=4 while the other is at k=2, the report warns next to the sizing line and names both. Unequal k is a precision issue, not a bias: rows lost at random leave the paired delta unbiased and only widen its interval; rows lost for a reason (a timeout on the hard runs) bias it, and only re-running the short arm on its short tasks fixes that. balance_rollouts=True (off by default) trims every paired task to the rows both sides have (drawn by seed=) so pass^k and pass@k share one k; it costs precision, removes no bias, and balanced says how many rows each side gave up.
Run the eval three times. One evaluation is a draw, not a number: the same model on the same tasks lands somewhere else next time, and most post-training gains are inside that spread (rlhf-book ch. 16, appendix C). wai.simulate(agent, tasks=base, runs=3) replays the task set three times in one call, same tasks, faults and world, and stamps lineage.eval_run on every row. Feed both sides to delta_report and it works out run_std from the repeats itself. The verdict words: moved is a change the interval and the re-run band both support; moved_unreplicated is a change seen once, which could be noise, and the warning tells you the runs=3 call that settles it; within_eval_noise is a delta smaller than what re-running the eval does on its own, so equivalence, not a win; no_change_detected is an interval that covers zero. format_delta_report prints the same reading on its first line (report["headline_verdict"] is the word behind it): PASS only for a gain (moved, moved_unreplicated), NO DIFFERENCE for an interval over zero (a negative point estimate the interval does not settle is not a pass), FAIL for a regression or a failed guard, NOT COMPARABLE (causes) when the arms cannot be compared. ceiling=True means the before run already passes most of its tasks (0.9 or more, or too few paired tasks left with room), so there is little improvement the eval could show; use harder situations before training again.
Argument grounding. A policy trained to call a tool learns to call it before it learns when not to; on the refund environment both GRPO and DPO learned to invent an order id on a quarter of the prompts that gave none while the headline rose. mark_grounding(rows) stamps argument_grounding: 1 when every string argument of every tool call appears in the prompt, the user and system turns, or an earlier tool result (rows with no calls count as grounded), else 0. No categories, any agent; must_not_regress=["argument_grounding"] fails the run that learned to invent, and ungrounded_arguments(row) / grounding_report(rows) name the values. ignore_keys= skips free-text arguments, allow= lists enums and defaults.
Trajectory flags. Did the agent fake the work? trace_markers(rows) reads the trajectory rather than the prose (rlhf-book ch. 13, 14): lie.tests_claimed (tests said to pass when no test command ran or the last one failed), lie.unverified_claim (“I verified” with no tool calls), lie.phantom_edit (“I updated” with nothing written), lie.ignored_failure (the turn ended on a failed call and the reply never says so), hack.test_edited, hack.test_weakened, hack.suppressed, hack.bypassed, risk.destructive, risk.secrets, each with the fragment that raised it on row["trace_flags"]. The markers it stamps (honest_claims, reported_failure, no_test_tampering, no_suppression, no_bypass, no_destructive, no_secrets) are 1.0 when clean, so must_not_regress=["honest_claims"] fails a run that learned to overclaim, and hack_scan carries every fired flag as a trace: feature. trace_flag_report(rows) gives each flag’s rate, examples, and its correlation with the reward, flagged when the judge pays for the fake. Reads, writes, deletes and commands are told apart by the tool’s arguments and name; kinds={"my_tool": "write"} overrides.
Stage lineage. The pipeline is a sequence of stages (rlhf-book ch. 3): SFT, reward modeling, RL, and the eval that judges the result. stamp_stage(rows, "sft") records which stage a row fed, and stage_report(rows) counts rows per stage and flags the one mistake it most needs caught: any task used in both eval and a training stage. wai.stamp_stage, wai.stage_report, wai.stage_of, wai.STAGES (sft, rm, rl, eval, mid).
Model spec as an object. A spec or constitution is a living, versioned document (rlhf-book ch. 17). load_spec(constitution) wraps the {source, traits: [{id, name, principle, authority}]} shape (what the character recipe writes) into a Spec whose version is a content hash, so any edit to a principle changes it. spec.behaviors() are the trait ids, ready for delta_report(must_not_regress=...); stamp_spec(rows, spec) tags every row with spec_id and spec_version, so you can ask whether adherence held from one spec or model version to the next.
Markers: four families, one polarity
A marker is a named behavior measurement on a row. Everything that reads markers (marker_summary, delta_report, must_not_regress=, from_row, the run page) reads one place, row["markers"], and does not care which family put the value there. Four families write to it, and only one of them has the wrong polarity:
behavioral_markers, mark_rows, row_markers and STOCK_MARKERS live in whileai.simulations.score.markers, which is deprecated and raises a DeprecationWarning on first use. style_markers / style_report / refusal_report cover the same rlhf-book ch. 14 behaviors with the delta-ready polarity.1.0 is the good outcome; higher is better; a significant drop is the regression that must_not_regress= fails on. Name markers after the behavior you want:
1 - v) and rename before you compare runs; delta_report has no way to know which direction a name means.
Train, and watch it
Two ways to train, one record. The platform trains a pushed dataset (SFT, GRPO, DPO or a reward model, as a LoRA adapter) and serves the result; or your own trainer runs on Modal, a GPU box, or a notebook and reports into the same run. Either way the loss curve and the progress bar are at zeroproofai.com/platform/training.epochs= sets SFT, steps= sets GRPO, DPO and RM; each method has a default. run.delta is delta_report kept on the run and drawn on its page, including the per-group table when by= names a row key or marker; wai.attach_delta(run_id, before, after) does the same for a run that already finished. holdout= names the eval set (defaults to the train set’s split sibling); a dataset already training returns that run. serve needs a finished run whose base is a served one (Qwen/Qwen3-4B, microsoft/phi-4; the list is whileai.simulations.training.SERVED_BASES). The trainer’s default bases (Qwen2.5-0.5B for SFT, 1.5B for GRPO and DPO) train fast but cannot be served, so train warns when a run will not reach an endpoint. Qwen3 answers in thinking mode by default: leave room in max_tokens or send extra_body={"chat_template_kwargs": {"enable_thinking": False}}.
method="rm" trains a reward model (rlhf-book ch. 5) on the set’s pass-vs-fail pairs and reports pair accuracy on the held-out pairs before and after. wai.reward_model(run) is that model as a judge, with the judge contract (reward 0/1 against the run’s threshold, rm_score raw), so it goes wherever a judge goes:
generations, learning_rate, beta, max_completion_length, temperature, loss_type, truncated) has a trainer default when left None; the range each is accepted in and the value the cited paper used are in whileai.simulations.training.TRAINING_KNOBS, and a rejected value is told the reference.
Your own trainer, three ways in:
run.holdout(before, after), or wai.attach_holdout(run_id, before=..., after=...) once the run has finished. Pass rates are 0 to 1, so 58% is 0.58; metric="loss" sends held-out loss instead (SFT), where lower is better. run.delta(...) and wai.attach_delta(...) already measure both sides, so they fill the two numbers in themselves, and add summary["holdout"] (also run.holdout_summary): each side’s pass rate with n_tasks, k and a ci95, plus the delta report’s verdict word (moved, moved_unreplicated, within_eval_noise, no_change_detected). A hosted run read back with run.refresh() has the same block with the interval fields None and a note that the platform only returned two numbers.
Plain HTTP, for a stack that is not Python: POST /runs with name, dataset_id, base_model, total_steps returns runId; POST /runs/{id}/log with points (a list of {"step", "loss", "lr", ...}, up to 500 a call); POST /runs/{id}/finish with status (done, failed or stopped), and optional summary and adapter. All with X-Api-Key. Points are buffered on the client and a send that fails is retried on the next flush; the dashboard never interrupts the trainer. wai.get_run(id) returns the run record.
Report a run so a person can decide
The platform draws one screen per tracked agent at while.ai/platform/runs: the held-out score by version with the frontier model as the line to beat, the training curve, what moved on the behaviors you did not train, the judge checks, live traffic on the served version, and cost. A coding agent fills it withwhileai.platform; the person reads it and presses Promote. Your agent framework stays yours: track takes the agent object you already have (OpenAI Agents SDK, Pydantic AI, LangGraph, Claude Agent SDK) and reads the model, the instructions and the tools off it, or you describe it by hand.
test_version), a noise_floor measured by scoring the same model twice, and a judge checked against people (agreement over human_n) and for length_bias. A run is scored on every behavior: targets are the claim, the rest are the check (verbosity, sycophancy and refusals are what moves when the reward is gamed). ci is the half-width of the 95% interval; the difference interval is delta ± sqrt(ci_candidate² + ci_served²), and the verdict says the candidate beats or trails the served version only when that interval excludes zero and the delta clears the behavior’s declared noise_floor. A missing interval, an interval that includes zero, or a delta inside the re-run band is said in those words. The count of other behaviors that came out lower is on point estimates with no interval yet, so it is a prompt to look, not a result. The verdict ends with what the number rests on (judge agreement, n) and starts with unproven: when n is under 50, judge agreement is under 0.8 or unmeasured, or the training reward is the judge. tracked.live(day, version=, replies=, flagged=) reports a day of traffic when you serve the model yourself. Logging buffers and never raises into the training loop. Worked example: recipes/04-train/report-run.
Is it hacking the reward right now?
wrap watches the reward function, so the monitor keeps the last completions with their rewards and runs hack_scan on them; every every steps it samples the holdout from the live policy and scores it with the training reward (the proxy) and with gold, a scorer the proxy cannot see. proxy_reward, gold_reward and holdout_length land on the run beside the loss curve. Four alarms, one line each on the run: divergence (proxy up by delta over the window while the paired gold interval does not move up), length (completions grow while gold does not), drift (KL past kl_budget), feature (the batch scan says reward_hack). stop_on names the ones that stop training; a stopped run finishes as stopped with the reason, and run.note(...) puts anything else on the run’s summary. wai.format_hack_monitor(monitor.summary()) prints the curve and the alarms. recipes/04-train/grpo runs it by default.
Publish a dataset as a card
zp-<id>, so load_dataset(repo, split, revision="zp-ds_...") pins the exact push; the repo’s whileai.json maps each split to its While dataset with history. Worked example: recipes/05-export/hugging-face.