Parameter reference
Every knobsimulate() takes. The defaults below are checked against the code by tests/api/test_readme_defaults.py, so a wrong number here is a failing test.
Aliases:
phrasings= / n= for requests_per_situation; repeats= for rollouts_per_request; unique= for unique_situations; policy= for system_prompt; risk= for fault_rate.
Experiment knobs
What a researcher changes between runs: who plays the user and how patient they are, how the three models sample, what the mock world answers, how hard the situations are, and what the search steers by. They go inadvanced={...}. data.report() carries every knob that produced the run, as resolved: the parameters above (mode, budget, seed, repeats, hard_share, fault_rate, strategy, dimensions, arm_weights, counts of tasks, traces and seeds, the grader name, simulator, agent_model, user_model, sampling and turn limits) and the advanced values below, so a saved run is its own experiment record. report()["fault_rate"] is the run’s rate; report()["world"]["default_fault_rate"] is the rate a fault plan with no rate of its own fires at, and world_note says so.
Engine internals
You should not need these. Every other number the engine uses is anadvanced key too, named after its field on whileai.simulations.defaults.RunKnobs, where a comment above each states why the default is what it is (a measurement, an rlhfbook.com chapter by URL, an arXiv id, or “convention, untested”). They are here so nothing in the engine is a number you cannot change, and so a report (data.report()["knobs"]) can say what a run ran under. A value outside its bounds is a ValueError that names the floor or ceiling and the default.
Output
Each row, indata.trajectories and on disk: prompt, messages, steps, final_text, scenario_id. Optional world_state, faults, reward, reason. llm_grade=True adds llm_reward. wai.rank(path) adds quality without changing reward. Every row also says how it was sampled: sampling is {"temperature", "max_tokens", "model"} as the model backend resolved them, or what you passed as simulate(sampling=...) for your own callable agent (None when you passed nothing, since only you know how it samples). policy_version names the model and the system prompt it ran under.
Three models can take part in a run, and by default they are one: the agent answers, and the same model writes the situations and plays the user in follow-up turns (only the judge is a different model). Every row says who did which job, next to model_version for the agent: writer_model (the writer’s model, or template, seed when no model wrote the prompt; a replay from tasks= or runs=N keeps the writer of the run it replays, since the situation was written once, and says lineage.replayed, plus lineage.replayed_from_run under runs=, so delta_report on two runs of one call sees one writer), user_model (absent when the agent took a single message), and judge_meta.model once graded; data.metadata and the .meta.json sidecar carry the same three. Every row also names the deploy prompt it was generated under: lineage.system_prompt_sha (the hash policy_version carries after @), lineage.system_prompt_head (its first 120 chars) and lineage.system_prompt_chars, with the full text once per run in data.system_prompts[<sha>], so a base rate measured under a full policy is never mistaken for one measured under a bare prompt; delta_report warns when its two arms differ on that hash. When the agent model also wrote the situations or played the user, data.degraded holds same_model and data.warnings says so in one sentence, with the fix: pass simulator= for the writer and user_model= for the user to put those jobs on a different model.
What goes to disk is the whole row, not a summary of it: data.rows (the same list output= and save() write, callable as data.rows() too) carries everything the trajectory carries, so a saved run can still prove its own provenance. That includes how the situation was drawn (scenario_dimensions, arm, selection_reason, behavior_signature, seed), who graded it and how that went (judge_name, judge_status, judge_meta, lineage, label_source), and what was measured on it (markers, read by marker_summary and delta_report). Two things never ship, at any depth of the row: the teacher-only privileged block and its principle / hidden_state / reference / rubric fields, which would put the answer key one step from a training file, and vector, the raw embedding the diversity search keeps in memory for the length of the run. A privileged block nested inside a carried field or a tool result is dropped the same way, before messages is rebuilt from the steps. data.trajectories keeps privileged in memory for the judge; so do the graded copies data.grade(judge=) returns.
pass@1, pass^k, pass@k
After grading,data.pass_at (also on the ScoredData from judge= and evaluate) gives pass@1, pass^k and pass@k off the same groups, one job each: pass@1 is the measurement headline (the agent runs once in production), pass^k is the reliability line (all k repeats pass), and pass@k minus pass@1 (.headroom) is what a grouped RL update has to learn from, the same asks group_signal counts as mixed. k is the smallest group of repeats; below repeats=4 the k-way numbers are None with a note rather than a noisy figure. With an LLM judge, pass@k inflates on false positives and pass^k on false negatives, so pass@1 stays the headline.
rl default allocates rollouts where groups split), k defaults to the smallest group and the line says which groups it left out; repeat_policy="fixed" gives every prompt the same k.
.per_task is a dict, {task: pass rate over that task's rollouts}, keyed by task_key(row) (the scenario_id, else task_id, else the prompt string), not indexed, so per_task[0] is a KeyError and not the first task. Iterate .per_task.items(); .per_task.values() is the pass-rate vector pass@1 averages.
Which of these carry an interval. All three pass numbers: pass_at(rows).ci95 is a bootstrap over tasks on pass@1, and pass_pow_k_ci95 / pass_at_k_ci95 bootstrap the per-group unbiased estimates over the k-eligible groups, so the reliability line is read with the uncertainty of the tasks behind it (rlhf-book ch. 16). Fewer than three groups gives None. What else carries one: metric_summary / marker_summary over markers, trace_flag_report over each trajectory marker’s clean share, refusal_report and judge_trust a Wilson interval, compare_runs / delta_report a bootstrap interval on the paired difference. If a number is not in that list and is not one of the three pass numbers, assume it is a point estimate.
Logprobs and off-policy checks
simulate(logprobs=True) records, on every agent turn, the summed log-probability of the tokens the policy generated and how many there were (step["logprob"], step["n_tokens"], totals on the row). A trainer that updates on these rollouts later needs that number to form the importance ratio exp(new_logprob - logprob); without it the update is off-policy and nothing says so. wai.logprob_report(rows) says how much was captured and whether reward tracks the policy’s confidence, which on a fair judge it should not. Score the same rows under a reference model, put its summed logprob in ref_logprob, and wai.mean_kl(rows) gives the sampled KL per generated token, overall and per task; wai.calibrate(rows, ref="ref_logprob") writes it into each row’s calibration.mean_kl. A turn the model cut at the token cap is marked truncated. Independently of logprobs, every agent step also records what its model call cost when the server reports it (step["input_tokens"], step["output_tokens"], summed into row["usage"]), which is what the platform counts per day.
staleness_report is the off-policy check (rlhf-book ch. 6): rows sampled by an older policy are usable only when they carry the sampler’s version and its logprobs, so the importance ratio can be formed; rows whose model_version differs from base_model are stale.
The judge on the row
The default judge is not the policy.wai.grade grades with hosted Phi-4 (WHILEAI_JUDGE overrides; any vllm:/openai: spec or a bare URL works), while rollouts come from hosted Qwen, because a judge grading its own model’s writing prefers it. When the judge and the rows’ model_version are the same model anyway, the grade report says so (self_judged, warnings).
A judge is a reward model, so two things ride with every label. Provenance: rows graded by wai.grade carry judge_name, judge_status, and judge_meta with the model, prompt hash, temperature, and version (<model>@<prompt sha>); a rubric edit is a new judge and the row says so. run_judge(version=...) records the same for your own judge. Accuracy: hand-label a sample into gold_reward and call wai.judge_agreement(rows) (or scored.agreement()) for agreement, Cohen’s kappa, the confusion counts, and pass_when_gold_fail, the gold failures the judge passed. Those are the rows a training run learns the failure from, so that rate matters more than the headline agreement. Pass a second scoring run as gold to measure the judge against itself. Fifty gold rows is the floor; the report says so below it.
Schema
Every row carriesschema_version ("1"). A row is a projection of four objects in whileai.simulations.schema: Task (the situation), Rollout (one episode), Judgment (a scorer’s verdict), Marker (a behavior measurement). wai.from_row(row) splits a row into them and wai.to_row(...) flattens them back. The wire contract is whileai/simulations/schemas/row-v1.json. Rows written before the stamp are version 0 and load by shape, so older files still work.