> ## Documentation Index > Fetch the complete documentation index at: https://docs.withwhile.com/llms.txt > Use this file to discover all available pages before exploring further. # whileai.simulations.score > Grading, judge checks, pass@k, selection for SFT and RL, hack scans. Grading, judge checks, pass\@k, selection for SFT and RL, hack scans. 99 public names. `import whileai.simulations as wai`, then `wai.name`. | Name | What it does | | --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`Criterion`](#criterion) | One rubric item. | | [`PassAt`](#passat) | pass\@1 / pass^k / pass\@k over graded groups. | | [`PublishGateError`](#publishgateerror) | The dataset must not be published as it stands. | | [`Rubric`](#rubric) | | | [`ScoredData`](#scoreddata) | Scored trajectories: the one representation grade and eval share. | | [`Spec`](#spec) | A versioned model spec. | | [`Trait`](#trait) | One named expectation. | | [`annotator_agreement`](#annotator_agreement) | How the annotators on `gold_labels` agree with each other. | | [`argument_grounding`](#argument_grounding) | 1.0 when every string argument of every tool call is grounded in the conversation (a rollout with no calls is grounded), else 0.0. | | [`attach_labels`](#attach_labels) | Write hand labels onto rows (in place) and return `(rows, report)`. | | [`attach_rubric`](#attach_rubric) | Put a rubric on each row's `privileged` block (in place). | | [`audit_grades`](#audit_grades) | Estimate the verifier's false-negative rate from a judged sample. | | [`behavior_signature`](#behavior_signature) | Hash of tool sequence, argument provenance, statuses, and outcome shape. | | [`behavioral_markers`](#behavioral_markers) | Rate of each stock marker over `rows` (fraction of rollouts that trip it). | | [`build_preference_pairs`](#build_preference_pairs) | Same-task chosen/rejected pairs for preference training (DPO-style). | | [`calibrate`](#calibrate) | Stamp `calibration` on every graded row, in place. | | [`classify_failure`](#classify_failure) | Fixed-vocabulary class for a failing row, from its reason and shape. | | [`compare_runs`](#compare_runs) | Is run `b` different from run `a` on `metric`? | | [`conduct_grade`](#conduct_grade) | Score agent conduct. | | [`coverage_gap`](#coverage_gap) | Which parts of an agent's policy the asks you already send never reach. | | [`curriculum`](#curriculum) | Split graded tasks into a training curriculum by measured difficulty. | | [`dataset_report`](#dataset_report) | One report a developer reads after simulate/grade: size, signal, mix. | | [`decontaminate`](#decontaminate) | Drop rows whose prompt overlaps an evaluation set (rlhf-book ch. | | [`dedupe_groups`](#dedupe_groups) | Drop repeat trajectories within one ask. | | [`delta_report`](#delta_report) | Compare `after` to `before` on pass\@1 and every shared marker. | | [`detectable_effect`](#detectable_effect) | The smallest gain `n_tasks` paired tasks can prove at `power`: `holdout_size` solved for the effect (`FIXED_POINT_STEPS` fixed-point steps, since the after-side variance depends on it). | | [`eval_variance`](#eval_variance) | How much an evaluation moves when the same model is evaluated again (rlhf-book ch. | | [`evaluate`](#evaluate) | Judge held-out rollouts under the exact contract `grade` uses. | | [`expected_outcome`](#expected_outcome) | What the checklist will look for on this task, in one sentence. | | [`filter_rl_rows`](#filter_rl_rows) | Split keep/drop. | | [`format_audit`](#format_audit) | The block a person reads: the summary, then the reasons. | | [`format_coverage_gap`](#format_coverage_gap) | The gap report as the block a person actually reads. | | [`format_curriculum`](#format_curriculum) | One-line-per-fact summary for a terminal. | | [`format_delta_report`](#format_delta_report) | The block a person reads: headline, then one line per metric. | | [`format_hack_scan`](#format_hack_scan) | The block a person reads: the regime, the floor, the ranking. | | [`format_hack_scan_diff`](#format_hack_scan_diff) | The block a person reads: what was learned, then the shifts. | | [`format_judge_trust`](#format_judge_trust) | | | [`format_leak_report`](#format_leak_report) | One line per fact, the summary first. | | [`format_markers`](#format_markers) | One line per marker, highest rate first. | | [`format_stages`](#format_stages) | | | [`grounding_report`](#grounding_report) | Over a row set: the share of rollouts with every argument grounded, the share with any call at all, and the most common invented values by tool and key, for a reviewer to look at. | | [`group_signal`](#group_signal) | Within-ask contrast. | | [`hack_scan`](#hack_scan) | Rank what separates reward within each ask against a permutation noise floor, and say what a grouped update would learn. | | [`hack_scan_diff`](#hack_scan_diff) | What the policy learned: the scan before training against the scan after, on rollouts scored by the same reward. | | [`holdout_size`](#holdout_size) | How many paired tasks a holdout needs to prove a gain of `effect`. | | [`judge_agreement`](#judge_agreement) | Agreement between the judge's `reward` and a trusted label. | | [`judge_pairs`](#judge_pairs) | Ask a judge which side of each pair is better, both ways round. | | [`judge_probes`](#judge_probes) | Try the reward hacks a policy finds first on the judge, on purpose. | | [`judge_trust`](#judge_trust) | The judge-trust report. | | [`leak_report`](#leak_report) | Which rows quote their own `privileged` block in the agent's text. | | [`length_report`](#length_report) | Truncated rollouts and asks whose reply lengths are far apart. | | [`load_spec`](#load_spec) | Build a `Spec` from a constitution dict, a list of traits, or a path to a JSON file with either shape. | | [`logprob_report`](#logprob_report) | Coverage and shape of the captured logprobs. | | [`mark_grounding`](#mark_grounding) | Copies of `rows` with `markers["argument_grounding"]` stamped, so `marker_summary`, `delta_report` and the run page read it. | | [`mark_rows`](#mark_rows) | Return copies of `rows` with the stock markers merged into each row's `markers` dict, ready for `marker_summary` / `delta_report`. | | [`marker_names`](#marker_names) | | | [`marker_summary`](#marker_summary) | `metric_summary` for every marker on the rows (or `names`). | | [`mean_kl`](#mean_kl) | Sampled `KL(pi \|\| pi_ref)` per generated token, overall and per task. | | [`metric_summary`](#metric_summary) | Mean over tasks with a task-bootstrap 95% interval. | | [`near_duplicate_prompts`](#near_duplicate_prompts) | Pairs of distinct asks whose token sets overlap at or above `threshold`. | | [`next_round`](#next_round) | The prompt set for the next round, from the last round's graded rollouts. | | [`normalize_judge_result`](#normalize_judge_result) | Coerce one judge return into the contract; never invent a reward. | | [`optimize`](#optimize) | One call after grading: concentrate for the post-training target. | | [`outcome_check`](#outcome_check) | (outcome, reason, checks). | | [`pairwise_judge`](#pairwise_judge) | A model judge for `judge_pairs`: `judge(a_row, b_row) -> \{"winner": "A" \| "B" \| "tie" \| None, "reason": str\}`. | | [`pass_at`](#pass_at) | pass\@1, pass^k and pass\@k from graded rows, grouped by task. | | [`preflight`](#preflight) | Spec-quality report for an agent. | | [`privileged_context`](#privileged_context) | The teacher's block for a row at birth: `hidden_state` (what the world knows that the ask does not say) and `reference` (what the checklist expects). | | [`publish_gate`](#publish_gate) | Check, calibrate, and report. | | [`rank_rows`](#rank_rows) | Score each row in place. | | [`recommend`](#recommend) | How much data this agent needs, from its own grid. | | [`reference_logprobs`](#reference_logprobs) | Stamp `ref_logprob` on every row: the reference's summed logprob over the tokens the policy generated. | | [`refusal_report`](#refusal_report) | Over-refusal on a benign set (rlhf-book ch. | | [`retire_solved`](#retire_solved) | Return the rows with every solved task removed. | | [`reward_correlations`](#reward_correlations) | corr(reward, feature) for the cheap features a judge tends to reward by accident: reply length, tool-call count, assistant turns, and the over-optimization signatures of rlhf-book ch. | | [`row_markers`](#row_markers) | The stock markers for one row's final text. | | [`rubric_judge`](#rubric_judge) | A judge for `run_judge` / `data.grade(judge=)` that scores the rubric item by item. | | [`rubric_of`](#rubric_of) | | | [`run_judge`](#run_judge) | Score trajectories with any judge. | | [`score_row`](#score_row) | Score one row. | | [`select_for_rl`](#select_for_rl) | Whole mixed groups up to roughly `target` rows. | | [`select_for_sft`](#select_for_sft) | Diverse correct demonstrations, at most `target` rows. | | [`spec_version`](#spec_version) | The content version of a Spec (or anything `load_spec` accepts). | | [`stage_of`](#stage_of) | The stamped stage, or None. | | [`stage_report`](#stage_report) | Rows per stage, tasks per stage, and the cross-stage leaks: any task used both in `eval` and in a training stage (sft/rm/rl/mid). | | [`staleness_report`](#staleness_report) | Which policies produced these rows, and can an update still use them. | | [`stamp_spec`](#stamp_spec) | Return copies of `rows` tagged with the spec they were produced or graded against: `spec_id` and `spec_version`. | | [`stamp_stage`](#stamp_stage) | Return copies of `rows` with `row["stage"] = stage`. | | [`style_markers`](#style_markers) | Stamp the style markers on every row's `markers` (in place) and return the rows. | | [`style_report`](#style_report) | How much of each signature the replies carry, and whether the reward pays for it. | | [`task_checklist`](#task_checklist) | Judge contract: conduct gated by the task's checkable outcome. | | [`task_key`](#task_key) | The one name every report groups a row's rollouts under. | | [`trace_flag_report`](#trace_flag_report) | How often each flag fires, what a reviewer should read, and whether the reward pays for it. | | [`trace_flags`](#trace_flags) | The flags that fire on one rollout: `{flag: evidence}`. | | [`trace_markers`](#trace_markers) | Stamp the trace markers on every row's `markers` (in place) and return the rows: 1.0 when the family is clean, 0.0 when a flag fired. | | [`trim_out_of_band`](#trim_out_of_band) | Difficulty band filter. | | [`trim_unanimous_groups`](#trim_unanimous_groups) | Drop asks whose k >= `min_k` rollouts all landed 0 or all landed 1. | | [`ungrounded_arguments`](#ungrounded_arguments) | The string arguments of the rollout's tool calls that appear nowhere in the context the call could draw on. | | [`write_rubrics`](#write_rubrics) | Draft one rubric per distinct prompt with a model and attach it to every row of that prompt (`privileged.rubric`, `source="model"`). | ## agreement Does the judge agree with labels you trust? ### judge\_agreement ```python theme={null} judge_agreement( rows: Sequence[dict], gold: str | Sequence[dict] = 'gold_reward', reward: str = 'reward', allow_model_gold: bool = False, ) -> dict[str, Any] ``` Agreement between the judge's `reward` and a trusted label. `gold` is either a key on the same rows (default `gold_reward`, the field `attach_labels` fills when you hand-label a sample) or a second row list from another scoring pass, matched by rollout id, scenario id plus rollout index, or prompt plus final text. Only exact 0/1 labels on both sides count; partial scores and unjudged rows are reported as skipped, not guessed. Returns `n`, `agreement`, `kappa` (Cohen, chance-corrected), the confusion counts, `pass_when_gold_fail` (the leak rate: gold failures the judge passed) and `fail_when_gold_pass`, both pass rates, `gold_kind` (where the labels came from: `"human"`, `"model"`, `"unknown"` for rows with no record), `ok` (rows were compared and the labels are a person's), and `warnings`. A second judge pass is model gold; rows with `gold_reward` but no `gold_kind` are unknown; either makes `ok` false with the reason unless `allow_model_gold=True`. ## audit Is the verifier failing answers that are right? ### audit\_grades ```python theme={null} audit_grades( rows: Sequence[dict], judge: Callable[[dict], Any], sample: int = 40, passes: int = 0, question: str = "The reference answer is the answer key for this task. Is the reply's final answer correct, meaning it gives the same result as the reference? Differences in formatting, rounding, ordering (unless the task asks for an order), column or field naming, or an equivalent way of expressing the same result do not make it wrong. A missing, different, or partial result does.", seed: int = 0, concurrency: int = 4, timeout: float = 120, judge_name: str | None = None, fn_warn: float = 0.1, ) -> dict[str, Any] ``` Estimate the verifier's false-negative rate from a judged sample. `rows` are graded by the verifier (`run_judge(rows, verifier)` or `data.grade(judge=verifier)`): `reward` 0/1, `reason` from the rule. `sample` failed rows (reward 0, judge ok) are drawn with `seed` and each is put to `judge` (any judge in the `run_judge` contract: `rubric_judge()`, `grade_llm`, your own callable) with the reference in place and `question` as the rubric when the row carries none. A judge reward at or above `PASS_THRESHOLD` (0.5) on a failed row is a false negative. `passes` samples passed rows the same way for the false-positive side. Returns `fn_rate` with `fn_ci95` (Wilson), `estimated_wrong_fails` (the rate over every failed row), `reasons` (the verifier's failure kinds in the sample, each with how many the judge overturned), a few `examples`, `fp_rate` when `passes` > 0, and `warnings`. Above `fn_warn` (`FN_WARN`, 0.10) the summary says to fix the verifier before training; `select_for_rl(audit=report)` and `optimize(audit=)` carry the same warning into the selection. ### format\_audit ```python theme={null} format_audit(report: dict[str, Any]) -> str ``` The block a person reads: the summary, then the reasons. ## checklist A task-specific checklist reward, derived from what the world knows. ### expected\_outcome ```python theme={null} expected_outcome(dims: dict | None) -> str | None ``` What the checklist will look for on this task, in one sentence. The same branches as `outcome_check`, read before the agent has acted. `None` when no rule applies (a compound ask). This is the teacher's `privileged.reference`: the answer key the student must never be shown. ### outcome\_check ```python theme={null} outcome_check(row: dict) -> tuple[int | None, str, dict[str, Any]] ``` (outcome, reason, checks). `None` when no rule applies to this task. ### privileged\_context ```python theme={null} privileged_context(dims: dict | None, faults: dict | None = None) -> dict ``` The teacher's block for a row at birth: `hidden_state` (what the world knows that the ask does not say) and `reference` (what the checklist expects). Empty when the task carries neither. ### task\_checklist ```python theme={null} task_checklist( row: dict, declared_tools: set[str] | None = None, ) -> dict[str, Any] ``` Judge contract: conduct gated by the task's checkable outcome. ## curriculum Curriculum: order tasks easy-to-hard and retire the solved ones. ### curriculum ```python theme={null} curriculum( rows: Sequence[dict], solved: float = 0.8, floor: float = 0.2, band: tuple[float, float] = (0.2, 0.8), tiers: int = 3, min_rollouts: int = 2, ) -> dict[str, Any] ``` Split graded tasks into a training curriculum by measured difficulty. A task is *solved* when its pass rate is above `solved` (retire it: an all-pass task is dead gradient). It is *not ready* when its pass rate is below `floor` (hold it: no signal until the policy can sometimes solve it). Everything from `floor` to `solved` inclusive is *trainable*, ordered easy to hard (highest pass rate first) and split into `tiers` difficulty buckets for a staged schedule. The defaults are the two edges of `DEFAULT_BAND` (20% and 80%), the same band `select_for_rl` keeps, so a task at 1 of 8 is not ready here and out of band there for the same reason. Tasks with fewer than `min_rollouts` graded rollouts cannot have a difficulty and are reported separately. Returns a report; nothing is mutated. `band` is recorded and used only to count how many trainable tasks sit in the reasoning-recipe 20-80% sweet spot, so you can see whether the set has usable signal. ### format\_curriculum ```python theme={null} format_curriculum(report: dict[str, Any]) -> str ``` One-line-per-fact summary for a terminal. ### retire\_solved ```python theme={null} retire_solved( rows: Sequence[dict], solved: float = 0.8, min_rollouts: int = 2, ) -> list[dict] ``` Return the rows with every solved task removed. A task above the `solved` pass rate teaches nothing, so its rollouts are dropped; tasks with too few rollouts to judge are kept. ## delta Did training move the behavior, and did anything else slip? ### delta\_report ```python theme={null} delta_report( before: Sequence[dict], after: Sequence[dict], target: str | None = None, must_not_regress: Sequence[str] = (), markers: Sequence[str] | None = None, by: str | Callable[[dict], Any] | None = None, run_std: float | Mapping[str, float | None] | None = None, run_std_runs: int | None = None, proxy: str | None = None, n_boot: int = 2000, seed: int = 0, balance_rollouts: bool = False, alpha: float = 0.05, power: float = 0.8, ceiling_pass_rate: float = 0.9, answered_gap_points: float = 0.1, answered_alpha: float = 0.01, ) -> dict[str, Any] ``` Compare `after` to `before` on pass\@1 and every shared marker. `alpha` is the false-positive rate every verdict runs at: each interval is at `1 - alpha` (`ci95` at the default), the re-run band uses the same quantile, and `family_error` is `1 - (1 - alpha) ** n_metrics`. `power` feeds the sizing line (`detectable_effect`, `holdout_size`). `ceiling_pass_rate`, `answered_gap_points` and `answered_alpha` are the flags' thresholds (`CEILING_PASS_RATE`, `ANSWERED_GAP_POINTS`, `ANSWERED_P_MAX`). The two sides should have the same number of rollouts per task. When a run lost rollouts (`data.report()["rollouts_lost"]`), one arm can sit at k=4 and the other at k=2; the report warns, next to the sizing line, naming both. Unequal k is a precision issue, not a bias: a task's pass rate is its mean over however many rows it has, so rows lost at random leave the paired delta unbiased and only widen its interval (simulated, k=4 against k=2 on half the tasks: mean delta on the true value, interval about 10% wider). Rows lost for a reason are the problem: a timeout that takes the hard runs, an empty reply on the long ones, and the surviving rows on that arm score higher than the arm does. No trimming fixes that; only re-running the short arm on its short tasks does, and `data.report()["rollouts_lost_by"]` says why the rows went missing. `balance_rollouts=True` (off by default) trims every paired task to the rows both sides have, chosen by `seed`, so pass^k and pass\@k share one k; it costs precision (another 10% on the interval in the same simulation) and removes no bias (failures dropped on one arm: delta 0.32 untrimmed, 0.32 trimmed, true 0.05), and `balanced` says how many rows each side gave up. `target` names the metric the run was meant to move (`"pass_at_1"` or `"marker:"`); the verdict on it is the headline. `proxy` names the metric the run was actually trained on (the training reward as a marker, e.g. `"marker:first_action"`). When the proxy moved up and the target did not, or the proxy's interval sits entirely above the target's, the report is `over_optimized` and fails: the policy learned something the target does not credit (rlhf-book ch. 14). `must_not_regress` lists metrics whose significant drop fails the report. Metric names for markers are the marker names; pass\@1 is `"pass_at_1"`. Tasks on one side only do not pair; their count is `n_unpaired_tasks` and, when any were dropped, a warning says so. `by` splits the target by a group on each row: a row key (top level, or a marker name) or a callable `row -> group`. The report gains `groups`: the target compared within each group, so a headline that moved cannot hide a kind of prompt that moved the other way. A group whose target dropped significantly is listed in `groups_down` and warned about; it does not flip `ok`, which stays the `must_not_regress` contract (name the group's metric there if it should). `run_std` is the evaluation's own re-run standard deviation (rlhf-book ch. 16, appendix C). Pass `eval_variance(...)["run_std_by_metric"]` so pass\@1 and each marker are judged against their own floor: a marker on a subset of tasks is several times noisier than pass\@1, and pass\@1's floor reads a re-run draw of it as a regression (#300). A scalar applies one floor to every metric, as before. A metric the mapping lacks, or carries as `None`, is never given another metric's floor: it gets `noise_note: "no_replicate_floor"`, a warning, and its verdict rests on the task interval alone. A metric whose delta is inside `noise_band(floor, n_a, n_b, df)` is `within_noise`: not improved, not slipped, not a regression, and a target there reads `within_eval_noise` rather than moved, because re-running the eval moves it that much on its own. The band is `floor * sqrt(1/n_a + 1/n_b)` (the delta is a mean of `n_a` runs against a mean of `n_b`) times 1.96 for a given floor, which is taken as the eval's spread. A floor that came from re-runs is an estimate, not the spread: pass `run_std_runs=` (how many re-runs it was computed from, `eval_variance(...)["n_runs"]`) and the band uses the two-sided t quantile at `df = run_std_runs - 1` instead (three re-runs: 4.30 x floor x sqrt(2) with one run per side, not 1.96; under pure noise the 1.96 band lets about one delta in five through at df=2). A given `run_std` without `run_std_runs` keeps 1.96 and a warning names the fix. When both row sets carry two or more `lineage.eval_run` values (`simulate(tasks=..., runs=3)`) the report computes each metric's floor itself, pooled over the two sides, and uses the t quantile at `df = sum(runs - 1)` instead (three runs per side: 2.78 x floor x sqrt(2/3)); `run_std` is the headline metric's floor, `run_std_by_metric` has them all, `noise_band` is the headline band, `noise_rule` spells it out, and `eval_runs` says how many runs each side had. With one run on either side and no `run_std` a target that moved reads `moved_unreplicated` and a warning says how to fix it: one evaluation is a draw, not a distribution (rlhf-book ch. 16, "why many comparisons are unreliable", and appendix C). `not_comparable` lists every reason the two arms cannot be compared at all (none are raised here; the comparability checks add theirs). `ceiling` is set when the before side already passes `CEILING_PASS_RATE` of its tasks, or when fewer than `CEILING_MIN_TASKS_WITH_ROOM` paired tasks (and under half) are not already passed every time: there is little room left for an improvement to show, whatever the training did. `config` says what each side was produced with (`pass_at(...).config` per side: task count, k, temperature, max\_tokens, policy and judge versions, prompt hash). A warning names each setting the two sides disagree on, and says so when both sides are the same policy version (rlhf-book ch. 16: a comparison is only as good as the settings it was run under). `config[side]["answered_share"]` is the share of rows per side with a spoken reply once `` markup is gone. Every rate is conditional on it. The two shares are compared with a pooled two-proportion z test; when it clears `ANSWERED_P_MAX` (p \< 0.01) the warning states p and the gap, and when the gap also exceeds the re-run band (or `ANSWERED_GAP_POINTS` with no band) the report fails with `answered` in `not_comparable` and names the mechanism: a reasoning base against a reasoning-suppressed adapter under one shared `max_tokens` runs out of budget inside `` and never answers, so the adapter wins every row the base did not reply to (#297). `not_comparable` lists every such cause under one prefix, `NOT COMPARABLE:`. `situations` is the cause when fewer than half the tasks are on both sides (`paired_share` under 0.5 with tasks on one side only): the arms drew different situation sets, so the delta over the few that pair is between two evals, and the fix is to pin the after side to the before run's tasks (`tasks=`) or compare per tier with `dataset_report`. A replay (`simulate(tasks=...)` or `runs=N`) keeps the writer of the run it replays on `writer_model`, so two runs of one call compare as one writer. Situations nobody's model wrote (a `seeds=` ask, the offline template writer, or a replay of either) count as one writer for this check: nothing there could have moved with the weights. `headline_verdict` is the verdict `format_delta_report` prints on its first line, in words: `PASS` only for a gain the report supports, `NO DIFFERENCE` for an interval over zero, `NOT COMPARABLE (causes)` when the arms cannot be compared, `FAIL` for a regression, a guard, or over-optimization. `ok` is the gate (no regression, no failed guard, comparable arms); it does not say the change helped. ### format\_delta\_report ```python theme={null} format_delta_report(report: dict[str, Any]) -> str ``` The block a person reads: headline, then one line per metric. ## grading Deterministic conduct rules for any tool-using agent. ### behavior\_signature ```python theme={null} behavior_signature(trajectory: dict) -> str ``` Hash of tool sequence, argument provenance, statuses, and outcome shape. ### conduct\_grade ```python theme={null} conduct_grade(trajectory: dict, declared_tools: set[str] | None = None) -> dict ``` Score agent conduct. Tool/sandbox faults are a flag, not a zero. ## grounding Argument grounding: did every tool argument come from the conversation? ### argument\_grounding ```python theme={null} argument_grounding(row: dict, **kwargs: Any) -> float ``` 1.0 when every string argument of every tool call is grounded in the conversation (a rollout with no calls is grounded), else 0.0. Keyword arguments are those of `ungrounded_arguments`. ### grounding\_report ```python theme={null} grounding_report(rows: Sequence[dict], **kwargs: Any) -> dict[str, Any] ``` Over a row set: the share of rollouts with every argument grounded, the share with any call at all, and the most common invented values by tool and key, for a reviewer to look at. ### mark\_grounding ```python theme={null} mark_grounding(rows: Sequence[dict], **kwargs: Any) -> list[dict] ``` Copies of `rows` with `markers["argument_grounding"]` stamped, so `marker_summary`, `delta_report` and the run page read it. ### ungrounded\_arguments ```python theme={null} ungrounded_arguments( row: dict, ignore_keys: Sequence[str] = (), allow: Sequence[str] = (), min_len: int = 3, ) -> list[dict[str, str]] ``` The string arguments of the rollout's tool calls that appear nowhere in the context the call could draw on. Each entry is `\{"tool", "key", "value"\}`. `ignore_keys` skips argument names that are free text by design (a note, a message body); `allow` lists values that are legal without appearing in the conversation (an enum, a default); strings shorter than `min_len` are skipped. ## hack\_scan What will the policy learn from this reward? Name it before training. ### format\_hack\_scan ```python theme={null} format_hack_scan(report: dict[str, Any], top: int = 12) -> str ``` The block a person reads: the regime, the floor, the ranking. ### format\_hack\_scan\_diff ```python theme={null} format_hack_scan_diff(report: dict[str, Any]) -> str ``` The block a person reads: what was learned, then the shifts. ### hack\_scan ```python theme={null} hack_scan( rows: Sequence[dict], endorsed: Sequence[str] = (), features: Mapping[str, Callable[[dict], float | None]] | None = None, auto: bool = True, top_k: int = 200, n_perm: int = 100, min_obs: int = 20, seeds: int = 12, reward: str = 'reward', seed: int = 0, top_features: int | None = 20, alpha: float = 0.05, ) -> dict[str, Any] ``` Rank what separates reward within each ask against a permutation noise floor, and say what a grouped update would learn. `alpha` (`ALPHA`, 0.05) sets the floor: `tau` is the `1 - alpha` quantile of the strongest feature's |rho| when reward is shuffled within ask, so a feature above it clears chance at that rate. `rows` are graded rollouts, several per ask (`mode="rl"`); the reward under `reward` may be 0/1 or partial credit. `endorsed` names the features the reward is supposed to track, as substrings of feature names (`"lookup_order"` matches `tool:lookup_order` and `contains:lookup_order`; `"marker:grounded"` a marker). Without it the scan still ranks and floors, but cannot call a hack a hack. `features` adds hand-tier columns: `{"name": lambda row: value}`. `top_features` caps the ranking in the report (`None` lists all). Returns `regime` (`train`, `reward_hack`, `pool_exhausted`, `no_signal`, `degenerate`, `unknown`), `tau` (the floor), `features` ranked by |within-ask correlation| with the pooled correlation beside each, `top_feature`, `endorsed_on_top`, `integrity` (share of the above-floor signal that sits on an endorsed feature), the support numbers (asks all-pass, all-fail, mixed, gradient capacity), and `warnings` in one line each. `degenerate` is the refusal: an ask holds fewer than `MIN_DISTINCT_PER_ASK` distinct rollouts at the median (`distinct_per_ask`) and two or more features sit at |rho| >= `DEGENERATE_RHO`, exactly collinear with reward and with each other because nothing else could happen at that variety. The ranking cannot separate them and the noise floor is no help (it tells signal from noise, not one perfect explanation from another), so `top_feature` and `integrity` are `None`, `inverted` is empty, no hack is claimed, and `collinear` lists the tied features. The direction is withheld with the name: at that variety an endorsed feature is negative exactly when it fell on the failing trajectory, so the sign is the same coin flip. Collinear features on a varied pool are left alone: there the ranking found two names for one behavior, and a genuinely inverted endorsed feature is still reported. ### hack\_scan\_diff ```python theme={null} hack_scan_diff( before: Sequence[dict], after: Sequence[dict], endorsed: Sequence[str] = (), top: int = 10, **scan_kwargs: Any, ) -> dict[str, Any] ``` What the policy learned: the scan before training against the scan after, on rollouts scored by the same reward. A feature that clears the floor after and did not before is what the update moved toward; one that dropped out is what it moved away from. `gained` and `lost` list them with both correlations, `moved` the largest shifts either way, and `learned` is the one line to read: the top gained feature, and whether it is endorsed. `scan_kwargs` reach both `hack_scan` calls. When either side comes back `degenerate`, every feature there is above the floor at |rho| 1 and no feature can be said to have gained it. `learned` says which side could not be read and why, and no hack is claimed; the rows are still listed so the shift is visible. ## hygiene Row hygiene a grouped RL update or a rejection-sampling pass cares about. ### dedupe\_groups ```python theme={null} dedupe_groups(rows: Sequence[dict]) -> tuple[list[dict], dict[str, Any]] ``` Drop repeat trajectories within one ask. Keeps the first of each (behavior signature, normalized reply) pair. Reports how many pairs disagreed on reward, which is label noise the judge introduced. ### length\_report ```python theme={null} length_report(rows: Sequence[dict], max_spread: float = 4.0) -> dict[str, Any] ``` Truncated rollouts and asks whose reply lengths are far apart. `max_spread` is max / median within one ask. ### near\_duplicate\_prompts ```python theme={null} near_duplicate_prompts( rows: Sequence[dict], threshold: float = 0.8, ) -> dict[str, Any] ``` Pairs of distinct asks whose token sets overlap at or above `threshold`. Quadratic in the number of asks; fine for the few thousand a run produces. Report only. ### reward\_correlations ```python theme={null} reward_correlations( rows: Sequence[dict], threshold: float = 0.3, ) -> dict[str, Any] ``` corr(reward, feature) for the cheap features a judge tends to reward by accident: reply length, tool-call count, assistant turns, and the over-optimization signatures of rlhf-book ch. 14 (boilerplate, hedging, sycophancy, refusal phrases, 1 when present; see `score.style`), plus every trajectory flag that fired on any row (`lie.*`, `hack.*`, `risk.*`; see `score.trace`). Any |corr| at or above `threshold` is flagged. A negative tool-count correlation means the reward pays the policy to do less; a positive phrase or flag correlation means it pays for the tic or the fake. ## judge\_trust Can the judge be trusted? The reward is only as good as the judge. ### format\_judge\_trust ```python theme={null} format_judge_trust(report: dict[str, Any]) -> str ``` ### judge\_probes ```python theme={null} judge_probes( rows: Sequence[dict], judge: Callable[[dict], Any], probes: str | Sequence[str] = 'all', rubric: str | None = None, sample: int = 40, seed: int = 0, concurrency: int = 8, flip_flag: float = 0.1, ) -> dict[str, Any] ``` Try the reward hacks a policy finds first on the judge, on purpose. Each probe in `probes` (`"all"` or names from `PROBES`) mutates up to `sample` graded rows one way and re-judges them. An additive probe (filler, the rubric's words, a success claim, the ask echoed, a sycophantic opener) reports `exploit_rate`: the share of originally failing replies that pass once the text is added. A replacement probe (a well-formed tool call with empty arguments, a refusal) reports the share of replies that pass with the content gone. `rubric` is the text the keyword probe draws words from; without it the row's system prompt is used. A probe that applies to no row is `skipped` with the reason. Returns per-probe counts and rates, `exploitable_by` (probes at or over `flip_flag`, `FLIP_FLAG` by default), and one warning per exploit. ### judge\_trust ```python theme={null} judge_trust( rows: Sequence[dict], judge: Callable[[dict], Any] | None = None, gold: str = 'gold_reward', sample: int = 40, seed: int = 0, concurrency: int = 8, probes: str | Sequence[str] | None = None, rubric: str | None = None, min_agreement: float = 0.8, min_kappa: float = 0.6, allow_model_gold: bool = False, length_gap_flag: float = 0.15, flip_flag: float = 0.1, ) -> dict[str, Any] ``` The judge-trust report. See the module docstring. The floors and flags are keywords with their defaults in `whileai.simulations.defaults`: `min_agreement` (0.8, the human-human agreement of MT-Bench, arXiv:2306.05685), `min_kappa` (0.6, Landis and Koch "substantial"), `length_gap_flag` (0.15) and `flip_flag` (0.10). `rows` carry the judge's `reward`; rows that also carry `gold` (0/1, default `gold_reward`) feed the agreement, held-out, and length checks. Pass `judge` to add the perturbation checks, which call it on up to `sample` rows twice more. `probes="all"` (or a list of names from `PROBES`) adds `judge_probes`, one more pass over the sample per probe; `rubric` feeds the keyword probe. `ok` is true only when a gold-labeled check ran against a person's labels, the Wilson lower bound of agreement reached `min_agreement`, kappa reached `min_kappa`, and nothing else was flagged. With no labels every check has `n=0`, so `ok` is false with a warning saying the judge is unmeasured, not failed. `gold_kind` in the report says where the labels came from; model or unknown gold makes `ok` false with the reason unless `allow_model_gold=True`. The keys a caller reads first: `ok` (measured and clean), `agreement["agreement"]` and `agreement["ci95"]` (the number and its Wilson interval, not `ci`), `agreement["n"]` (labels compared), `gold_kind`, and `warnings`, where every line names its own fix. `format_judge_trust(report)` prints the whole thing. ## judging One judge contract for grading and evaluation, and the loop around it. ### ScoredData ```python theme={null} ScoredData( rows: list[dict], run_id: str, source: str, judge_name: str, model: str | None = None, ) ``` Scored trajectories: the one representation grade and eval share. Iterates as plain dicts, so it feeds `simulate(traces=...)`, `mine_traces`, `export_training` and JSONL writers directly — no conversion scripts. `.rows` is a `RowList`: a list that also answers to being called, so both `scored.rows` and `scored.rows()` give the scored rows. `SimulationData.rows`, what `simulate()` returns, behaves the same way, so the two spellings are interchangeable across `simulate() -> run_judge()`. `.warnings` is the list of hollow-run notes `run_judge` filled; print it before reading any number. #### ScoredData.agreement ```python theme={null} agreement( self, gold: str | Sequence[dict] = 'gold_reward', reward: str = 'reward', ) -> dict[str, Any] ``` Agreement of this run's rewards with a trusted label. See `judge_agreement`. #### ScoredData.failed\_traces ```python theme={null} failed_traces(self) -> list[dict] ``` Failures, ready to hand to `simulate(traces=...)`. #### ScoredData.partials ```python theme={null} partials(self) -> list[dict] ``` Rows with a continuous reward strictly between 0 and 1. The scalar lane: 1 pass, 0 fail, partials here, None unjudged - every contract-legal reward is visible in exactly one view. #### ScoredData.push ```python theme={null} push(self, name: str, **kwargs: Any) -> dict ``` Upload the scored rows to the platform: `push_rows(self.rows, name, ...)`. Same keywords as `push_rows` (`gate=`, `mode=`, `agent=`, `purpose=`, `parent=`, `endorsed=`, `strict_hacks=`). The graded copy is what a gated RL push needs, and `SimulationData.push` cannot see it: `grade(judge=)` leaves the run's trajectories ungraded on purpose. #### ScoredData.select\_by\_reward\_range ```python theme={null} select_by_reward_range( self, lo: float, hi: float, inclusive: bool = True, ) -> list[dict] ``` Rows whose numeric reward falls in \[lo, hi] (or (lo, hi)). #### ScoredData.select\_for\_preference ```python theme={null} select_for_preference( self, max_pairs_per_prompt: int = 1, min_margin: float = 1.0, length_match: bool = True, ) -> tuple[list[dict], dict[str, Any]] ``` Chosen/rejected pairs from same-task contrast. Failures earn here. #### ScoredData.select\_for\_rl ```python theme={null} select_for_rl( self, target: int = 1000, lo: float = 0.2, hi: float = 0.8, has_tools: bool = True, ) -> tuple[list[dict], dict[str, Any]] ``` Whole mixed-reward groups for RL; groups never split. `lo` and `hi` default to `DIFFICULTY_BAND` (0.2, 0.8), the same band `select_for_rl` and `optimize` use; they used to be 0.3 and 0.7 here alone. #### ScoredData.select\_for\_sft ```python theme={null} select_for_sft(self, target: int = 1000) -> tuple[list[dict], dict[str, Any]] ``` Diverse correct demonstrations: 1-labeled, deduped by behavior. #### ScoredData.unjudged ```python theme={null} unjudged(self) -> list[dict] ``` Rows the judge could not score. Never treated as failures. ### build\_preference\_pairs ```python theme={null} build_preference_pairs( rows: Sequence[dict], max_pairs_per_prompt: int = 1, min_margin: float = 1.0, length_match: bool = True, ) -> tuple[list[dict], dict[str, Any]] ``` Same-task chosen/rejected pairs for preference training (DPO-style). A pair exists only where the same prompt has two trajectories whose rewards differ by at least `min_margin` — the contrast is the training signal, so failures are supply here, not waste. The default `1.0` pairs 1-labeled with 0-labeled rows only; `0.5` also admits partial-credit rows against a full pass or fail. Rows without a valid judge result never pair. Each pair keeps what the trainer and the reviewer need to trust it: * `chosen_score` / `rejected_score` / `margin`: the raw scores and their gap, so a margin-aware loss (Llama 2 style) can use them and a reviewer can see how far apart the two really are. * `chosen_model` / `rejected_model` / `same_policy`: which policy produced each side. Preference data works best when both sides come from the policy being trained (Tulu 3, rlhf-book ch. 11); a mixed pair is still a pair, but it is labeled as off-policy. * `length_delta`: chosen reply chars minus rejected. DPO exploits a length gap faster than it learns the behavior (rlhf-book ch. 8), so with `length_match=True` each chosen row takes the rejected row closest to it in length, and the report says how often chosen is still the longer side. Returns (pairs, report); each pair carries both parents' lineage. ### evaluate ```python theme={null} evaluate( rows: Sequence[dict] | Any = None, judge: Callable[[dict], Any] | None = None, grader: Callable[[dict], Any] | None = None, model: str | None = None, eval_set: Sequence[Any] | None = None, judge_name: str | None = None, run_id: str | None = None, concurrency: int = 8, timeout: float | None = None, scale: tuple[float, float] | None = None, tools: Sequence[dict] | Sequence[str] | None = None, ) -> ScoredData ``` Judge held-out rollouts under the exact contract `grade` uses. Read the result's `warnings` before its numbers: no rollout called a tool, a declared tool none touched (`tools=`, or pass the `SimulationData` as `rows`), a marker that fired on no row. Same engine, same schema; only the lineage source differs. Feeding `evaluate(...).traces` to `simulate(traces=...)` is the loop-closing move. `grader=` is the doctrine-sketch name for the judge callable; either spelling works, not both. `eval_set=` (prompt strings or rows) checks that the rollouts actually cover the frozen evaluation set and reports the gap on the result's `eval_coverage` instead of letting a silent partial eval pass as a full one. ### normalize\_judge\_result ```python theme={null} normalize_judge_result( raw: Any, scale: tuple[float, float] | None = None, ) -> dict[str, Any] ``` Coerce one judge return into the contract; never invent a reward. `scale=(lo, hi)` reads the judge's number as a rating on that scale (a 1 to 5 Likert, a 0 to 10 score): the row's `reward` is the rating mapped onto \[0, 1] and `judge_meta` keeps `rating` and `scale`. A rating outside the scale is a contract break, as a reward outside \[0, 1] is without one. A dict may carry the number as `rating` instead of `score` when a scale is set. ### run\_judge ```python theme={null} run_judge( rows: Sequence[dict] | Any, judge: Callable[[dict], Any], judge_name: str | None = None, source: str = 'grade', model: str | None = None, run_id: str | None = None, concurrency: int = 8, timeout: float | None = None, version: str | None = None, scale: tuple[float, float] | None = None, tools: Sequence[dict] | Sequence[str] | None = None, ) -> ScoredData ``` Score trajectories with any judge. Originals are left unmodified. `tools=` is the agent's declared tool list (or names); with it the result's `warnings` also say which declared tools no rollout called. Passing the `SimulationData` itself as `rows` supplies it. Each scored row is a copy of the input row plus `reward`, `reason`, `judge_status`, `judge_meta`, and a `lineage` record naming the scoring run, its source (grade or eval), the judged model, and the parent trajectory. Rows whose judge result breaks the contract keep `reward=None` and a non-ok status; they are counted, not hidden. `version` names the judge's version (model, prompt hash, whatever would change its labels); it lands in `lineage.judge_version` and reads back as `Judgment.scorer.version`. ## labels Human labels on rows: who said what, and do they agree. ### annotator\_agreement ```python theme={null} annotator_agreement(rows: Sequence[dict]) -> dict[str, Any] ``` How the annotators on `gold_labels` agree with each other. `per_annotator`: labels given and pass share. `multi_labeled`: rows with two or more annotators; `unanimous` the share of those where every label matched; `kappa` Cohen's kappa when exactly two annotators labeled the same rows (`pair` names them), else `None`. `disagreements` lists the split rows (prompt, labels) so a person can read the ones the guideline did not settle (rlhf-book ch. 11). ### attach\_labels ```python theme={null} attach_labels( rows: Sequence[dict], labels: Any, annotator: str | None = None, kind: str = 'human', replace: bool = False, ) -> tuple[list[dict], dict[str, Any]] ``` Write hand labels onto rows (in place) and return `(rows, report)`. `labels` is a JSONL path, a list of dicts, or a `{key: label}` mapping. A dict label names its row by `key` / `rollout_id`, `scenario_id` + `rollout_index`, or `prompt` (+ `final_text`), and carries `label` (or `reward` / `gold_reward`: 0 or 1), and optionally `annotator`, `note`, `ts`. `annotator` here is the default for labels that name none. `kind` is recorded on each label (`"human"`; `"model"` for a stronger model's labels). Each row gains `gold_labels` (every label, appended unless `replace`), `gold_reward`, the majority of its labels, and `gold_kind`: `"human"` when every label on the row is a person's, else the other kind. `judge_trust` and `judge_agreement` only count human gold as a measurement of the judge. A tie leaves both unset. Labels that name no row, or carry no 0/1 value, are counted and listed. ## logprobs What the policy's own log-probabilities buy you. ### logprob\_report ```python theme={null} logprob_report(rows: Sequence[dict]) -> dict[str, Any] ``` Coverage and shape of the captured logprobs. `mean_token_logprob` is total logprob over total tokens. The per-row quantiles are of each row's own mean, so one long rollout does not dominate. `corr_reward_confidence` is Pearson between the 0/1 reward and the per-row mean over graded rows: a strong positive value says the judge rewards fluency, not behavior. ### mean\_kl ```python theme={null} mean_kl( rows: Sequence[dict], ref: str | Sequence[dict] = 'ref_logprob', ) -> dict[str, Any] ``` Sampled `KL(pi || pi_ref)` per generated token, overall and per task. `ref` is either a key on the same rows holding the reference model's summed logprob over the same tokens (default `ref_logprob`), or a second row list scored under the reference, matched by rollout id, scenario id plus rollout index, or prompt plus final text, carrying `logprob`. Rows missing either side are skipped and counted. Per task the estimate pools tokens across that task's rollouts, which is what a per-task difficulty record wants. ### staleness\_report ```python theme={null} staleness_report( rows: Sequence[dict], base_model: str | None = None, ) -> dict[str, Any] ``` Which policies produced these rows, and can an update still use them. rlhf-book ch. 6 (asynchronous RL, truncated importance sampling): rows sampled by an older policy are usable when the row carries the sampler's version and its logprobs so the ratio can be formed; rows from an unknown sampler are not. `versions` counts rows per `policy_version` (`model_version` when the row predates it); `base_model` names the model about to be trained, and rows whose `model_version` differs are `stale`. Coverage says how many rows carry `sampling`, `logprob` and `token_logprobs`. ## markers Stock behavioral markers for the over-optimization signatures (book ch. 14). ### behavioral\_markers ```python theme={null} behavioral_markers( rows: Sequence[dict], names: Sequence[str] | None = None, ) -> dict[str, float] ``` Rate of each stock marker over `rows` (fraction of rollouts that trip it). The over-optimization dashboard in one call. Deprecated: use `score.style.style_report` for the delta-ready view. ### format\_markers ```python theme={null} format_markers(report: dict[str, float]) -> str ``` One line per marker, highest rate first. ### mark\_rows ```python theme={null} mark_rows( rows: Sequence[dict], names: Sequence[str] | None = None, extra: dict[str, Callable[[dict], float]] | None = None, ) -> list[dict] ``` Return copies of `rows` with the stock markers merged into each row's `markers` dict, ready for `marker_summary` / `delta_report`. `extra` adds custom named detectors `row -> value`. Existing marker values are kept; stock names overwrite only themselves. Deprecated: presence polarity (1 = tic present) reads a `delta_report` paired comparison backwards. Use `score.style.style_markers`. ### row\_markers ```python theme={null} row_markers(row: dict, names: Sequence[str] | None = None) -> dict[str, int] ``` The stock markers for one row's final text. ## optimize Concentrate a big simulated batch into the dataset post-training needs. ### filter\_rl\_rows ```python theme={null} filter_rl_rows( rows: Sequence[dict], has_tools: bool = True, ) -> tuple[list[dict], dict[str, Any]] ``` Split keep/drop. Does not mutate `rows`. ### group\_signal ```python theme={null} group_signal( rows: Sequence[dict], lo: float = 0.2, hi: float = 0.8, ) -> dict[str, Any] ``` Within-ask contrast. Signal is a group whose k rollouts disagree. A grouped RL update learns from a mix of 0 and 1 on the same ask, ideally with pass rate p in \[`lo`, `hi`]. Unanimous groups are dead gradient. Groups of one rollout cannot mix and are counted separately, not blamed. ### next\_round ```python theme={null} next_round( prior: Sequence[dict], tasks: Sequence[Any] | None = None, lo: float = 0.2, hi: float = 0.8, ) -> dict[str, Any] ``` The prompt set for the next round, from the last round's graded rollouts. A round trained on the file it started from keeps paying for groups that give no gradient: at a 0.65 training reward about half the groups are all-pass or all-fail. The band is the fix the book already names (rlhf-book ch. 7: filter to the 20-80% band; ch. 6, DAPO's dynamic sampling drops groups with no contrast), applied to what the *current* policy does rather than what the base did. `prior` is round N's graded rollouts (`simulate(tasks=..., repeats=k)` on the round-N policy, or the trainer's own sampled rows); each task's pass rate over them decides: inside `[lo, hi]` it is kept, above `hi` it is solved and dropped, below `lo` it is unsolved and dropped. `tasks` restricts the candidates (rows, task dicts with a `prompt`, or prompt strings); a task with no prior rollouts is `unknown` and kept, since nothing says it is flat. Returns `tasks` (one representative row per kept task: the prior row, with `calibration.pass_rate` and the band), the counts `kept`, `dropped_solved`, `dropped_unsolved`, `unknown`, `pass_rates` per task, `band`, `from_policy` (the policy versions the prior rows came from) and `prompt_set_sha`: the identity of the kept set, for lineage on the run. Push the kept rows as the next train set with `parent=` the last one. ### optimize ```python theme={null} optimize( source, mode: str | None = None, target: int = 1000, output: str | None = None, band: tuple[float, float] = (0.2, 0.8), enforce_band: bool = True, select: str = 'top_per_prompt', min_reward: float = 1.0, endorsed: Sequence[str] = (), truncated: str = 'drop', order: str = 'spread', audit: dict[str, Any] | None = None, ) -> tuple[list[dict], dict[str, Any]] ``` One call after grading: concentrate for the post-training target. `source` is a `SimulationData`, a row list, or a JSONL path. `mode` defaults to the data's own mode: `"sft"` picks diverse correct demonstrations (`select` and `min_reward` as in `select_for_sft`), anything else keeps whole mixed RL groups inside the difficulty `band` (default 20%-80% pass rate; `enforce_band=False` only ranks out-of-band asks last; `order` is `"spread"` across pass rates or `"middle"` first, see `select_for_rl`). `endorsed` names what the reward should track (feature-name substrings such as `"tool:lookup_order"`), so the RL report's `hack_scan` can call a shortcut a hack. Returns `(rows, report)`; writes `output` when given, or `..jsonl` next to a path source. Never overwrites the source file unless `output` names it explicitly. ### recommend ```python theme={null} recommend( tools: Sequence[dict] | None = None, policy: str = '', system_prompt: str | None = None, mode: str = 'sft', target: int | None = None, mixed_rate: float = 0.5, ) -> dict[str, Any] ``` How much data this agent needs, from its own grid. No guessing. `system_prompt=` is the same text under `simulate`'s spelling; `policy=` and `system_prompt=` are interchangeable here as there. Grounded two ways: the agent's measured covering grid (every cell wants `SATURATION_COPIES` visits, and selection wants surplus to choose from), and published post-training practice (curated agent SFT lands at 500 to 2,000 trajectories: FireAct 500, LIMA 1,000, AgentTuning 1,866; agent RL uses 8 to 16 rollouts per prompt and drops all-pass/all-fail groups: DAPO 2025, Skywork-OR1 2025). Returns the numbers plus `simulate_kwargs` ready to splat, and `reasoning` lines that show the arithmetic. ### select\_for\_rl ```python theme={null} select_for_rl( rows: Sequence[dict], target: int = 1000, lo: float = 0.2, hi: float = 0.8, enforce_band: bool = True, has_tools: bool = True, dedupe: bool = True, drop_truncated: bool = True, endorsed: Sequence[str] = (), truncated: str = 'drop', order: str = 'spread', prior: Sequence[dict] | None = None, audit: dict[str, Any] | None = None, ) -> tuple[list[dict], dict[str, Any]] ``` Whole mixed groups up to roughly `target` rows. Groups never split. `audit` is an `audit_grades` report on these rows' verifier; when it found the verifier rejecting right answers more than `FN_WARN` of the time, `hygiene_warnings` says to fix the verifier before training on the selection (#255). `prior` is the previous round's graded rollouts: tasks the round-N policy already solves (pass rate above `hi` on `prior`) or never solves (below `lo`) are dropped before anything else, so round N+1 trains on what that policy gets right 20-80% of the time rather than on the file round 1 started from (`next_round`; rlhf-book ch. 7). The report's `prior` block counts kept, dropped\_solved, dropped\_unsolved and unknown. `truncated` says what happens to a rollout cut at the token cap (rlhf-book ch. 6, DAPO's overlong handling; ch. 7 overlong filtering): `"drop"` removes it (the default; `drop_truncated=False` is the old spelling of `"keep"`), `"keep"` leaves it in with `overlong=True` and its own reward, riding with its ask rather than deciding it (the ask is unanimous, in band and ranked exactly as under `"drop"`, so `"keep"` never returns fewer rows than `"drop"`; a cut rollout's reward is not the contrast an ask is kept for), and `"penalize"` keeps it as a failure that does count: reward 0, the judged score under `reward_before_penalty`, so running past the cap is a negative signal instead of a rollout that vanished. A conduct-grade advisory 0.5 for truncation is unusable under `"keep"` and a 0 under `"penalize"`. After the row gates, duplicate and truncated rollouts (`dedupe`, `truncated`), the unanimous trim, and (`enforce_band`) the difficulty band, remaining asks are taken round-robin across observed fault kinds, so the dataset keeps a grounded spread of no-fault, miss, timeout, and already-done situations rather than one over-represented failure. Within a fault kind, `order="spread"` (the default) takes asks round-robin across their pass rates, so a 25% ask, a 50% ask and a 75% ask are picked in turn with no preference for the middle (rlhf-book ch. 7 filters to the 20-80% band and stops there; nothing in it says 50% is better than 30%). `order="middle"` is the older ranking by closeness to a 50% pass rate. The last group may overshoot `target`; an RL update wants the complete group or none of it. `enforce_band=False` keeps out-of-band asks and only ranks them last. The report's `hack_scan` block is the reward-hack scan over the selection (`hack_scan`: what separates reward within an ask, against a permutation floor; `endorsed` names what it should be), `correlations` the older pooled scan. Reward tracking a shortcut is a judge problem, flagged in `hygiene_warnings`, not pruned. Selected rows are stamped in place with the `calibration` measured on the rows as they arrived, before dedupe and the trims: the pass rate over the k repeats the grader saw is the task's difficulty, and re-measuring it on the survivors would report the post-dedup k under that name. `publish_gate` keeps the carried stamp. The k-way reliability numbers do not survive the prune, and `hygiene_warnings` says so when they were available before it. ### select\_for\_sft ```python theme={null} select_for_sft( rows: Sequence[dict], target: int = 1000, select: str = 'top_per_prompt', k: int | None = None, min_reward: float = 1.0, seed: int = 0, ) -> tuple[list[dict], dict[str, Any]] ``` Diverse correct demonstrations, at most `target` rows. Imitation clones what it sees, so only rows whose reward reaches `min_reward` (default 1.0: judge-approved) and that are not junk qualify; unanimity is not a problem here. A grader with partial credit ranks by its score: lower `min_reward` to admit it. `select` is the rejection-sampling rule (rlhf-book ch. 9, "Scoring Completions"): `"top_per_prompt"` keeps each prompt's highest-reward completion and then round-robins across behavior signatures (tool sequence, argument provenance, outcome shape) so every distinct way of being right appears before any repeats; `"top_k_overall"` keeps the `k` highest-reward completions across all prompts, several per prompt allowed; the two `random_*` rules are the book's control (same counts, seeded random picks) so a claimed gain from selection can be checked against chance. `k` defaults to `target`. ### trim\_out\_of\_band ```python theme={null} trim_out_of_band( rows: Sequence[dict], lo: float = 0.2, hi: float = 0.8, min_k: int = 2, ) -> tuple[list[dict], dict[str, Any]] ``` Difficulty band filter. Nothing to do with topic or relevance. "Out of band" here means outside the *difficulty* band `[lo, hi]` (default `DEFAULT_BAND`, 0.2 to 0.8): an ask is dropped when its pass rate over k >= `min_k` rollouts is too high (the policy almost always solves it) or too low (it almost never does), because either way it carries little gradient per rollout. It does not read the prompt, the topic, or the tools; a perfectly on-topic ask is dropped for being too easy, and an off-topic one the policy passes half the time is kept. Junk rows are a separate filter (`is_incomplete_junk`, applied by `optimize`), and nothing here filters by topic at all. Unanimous asks are `trim_unanimous_groups`'s job and are left alone here; singles always stay. ### trim\_unanimous\_groups ```python theme={null} trim_unanimous_groups( rows: Sequence[dict], min_k: int = 2, ) -> tuple[list[dict], dict[str, Any]] ``` Drop asks whose k >= `min_k` rollouts all landed 0 or all landed 1. The basic optimizer from the working decision: trim zeros and ones from tasks, then rerun the simulator and check the variance. Groups smaller than `min_k` (unique-situation runs) always stay; trimming them would gut an explore dataset, and they carry no group gradient either way. ## pairwise Pairwise judging: which of two replies to the same request is better, asked both ways round. ### judge\_pairs ```python theme={null} judge_pairs( pairs: Sequence[dict], judge: Callable[[dict, dict], Any] | None = None, spec: str | None = None, swap: bool = True, concurrency: int = 8, api_key: str | None = None, examples: int = 10, position_flip_flag: float = 0.2, prefers_rejected_flag: float = 0.2, ) -> tuple[list[dict], dict[str, Any]] ``` Ask a judge which side of each pair is better, both ways round. `judge(a_row, b_row)` returns `\{"winner": "A" | "B" | "tie", "reason"\}` (or that JSON as a string); without one the hosted model judge from `pairwise_judge(spec)` is used. With `swap=True` each pair is judged twice with A and B exchanged; a pair the judge decides differently in the two orders is recorded as a tie with `position_consistent=False`. Writes on each pair (in place, and returned): `pairwise` with `winner` (`"chosen"` | `"rejected"` | `"tie"` | `None` when the judge failed), `position_consistent`, `reasons`, `judge`; and `tie` (bool). Report: `position_flip_rate` (position bias: the judge's answer changed with the order), `tie_rate`, `agrees_with_scores` (the pairwise winner is the pointwise `chosen`), `prefers_rejected` (the two disagree outright, the rows a person should read), `failed`. A `position_flip_rate` at or over `position_flip_flag` (`POSITION_FLIP_FLAG`, 0.2: Zheng et al. arXiv:2306.05685 measured 35% of GPT-4 verdicts flipping with the order) and a prefers-rejected share at or over `prefers_rejected_flag` each add a warning. ### pairwise\_judge ```python theme={null} pairwise_judge( spec: str | None = None, api_key: str | None = None, prompt: str | None = None, policy: str = '', tools: Sequence | None = None, timeout: float = 120, max_tokens: int = 120, request_chars: int = 4000, ) -> Callable[[dict, dict], Verdict] ``` A model judge for `judge_pairs`: `judge(a_row, b_row) -> \{"winner": "A" | "B" | "tie" | None, "reason": str\}`. `spec` is a backend spec (default the hosted judge); `prompt` replaces the pairwise system prompt. The judge's name is `@` so a prompt edit is a new judge. `max_tokens` is the judge's reply budget and `request_chars` how much of the request it is shown. ## passat pass\@1, pass^k and pass\@k from the same graded groups. ### PassAt ```python theme={null} PassAt( k: int, pass_at_1: float | None, pass_pow_k: float | None, pass_at_k: float | None, n_groups: int, n_rows: int, n_groups_at_k: int = 0, n_groups_imputed: int = 0, per_task: dict[str, float] = , note: str = '', config: dict[str, Any] = , ci95: tuple[float, float] | None = None, pass_pow_k_ci95: tuple[float, float] | None = None, pass_at_k_ci95: tuple[float, float] | None = None, ) -> None ``` pass\@1 / pass^k / pass\@k over graded groups. See module docstring. Every field, and the name it prints as in `str(...)`. The printed line and the attribute are not spelled the same: pass^k is `pass_pow_k` (not `pass_hat_k`), and the print says both once so the attribute is readable off it. | attribute | prints as | what it is | | ------------------ | ---------------- | --------------------------------------- | | `k` | `k=4` | draw size the k-way numbers used | | `pass_at_1` | `pass@1` | mean per-task pass rate, the headline | | `pass_pow_k` | `pass^4` | chance all k repeats pass (reliability) | | `pass_at_k` | `pass@4` | chance at least one of k passes | | `headroom` | `headroom` | property: pass\@k minus pass\@1 | | `ci95` | `[lo..hi]` | task-bootstrap interval on pass\@1 | | `pass_pow_k_ci95` | `[lo..hi]` | same for pass^k; `None` under 3 groups | | `pass_at_k_ci95` | `[lo..hi]` | same for pass\@k; `None` under 3 groups | | `n_groups` | `N groups` | tasks pass\@1 averaged over | | `n_rows` | not printed | graded rows behind those tasks | | `n_groups_at_k` | not printed | tasks the k-way numbers used | | `n_groups_imputed` | not printed | short unanimous tasks counted in | | `per_task` | not printed | `{task key: pass rate}`, a dict | | `note` | tail of the line | why a number is missing, and the fix | | `config` | token-cap share | how the rows were made (`run_config`) | `to_dict()` uses these same keys, with `headroom` added and the intervals as lists. ### pass\_at ```python theme={null} pass_at( rows: Sequence[dict] | Any, k: int | None = None, min_k: int = 4, unanimous_short: bool = False, ) -> PassAt ``` pass\@1, pass^k and pass\@k from graded rows, grouped by task. A task is a situation, not a string. Rows group under `task_key`: the engine's `scenario_id` when the row has one, else `task_id`, else the prompt text. In `mode="rl"` the repeats of one opener share a `scenario_id`, and so do the textured phrasings of one situation, so those phrasings pool into one task on purpose: the question is whether the agent handles the situation, not one wording of it. `compare_runs`, `delta_report`, `eval_variance`, `curriculum` and `group_signal` count tasks with the same key, so `pass_at(rows).n_groups` and `delta_report(...)["n_paired_tasks"]` agree on the same rows. Only binary `reward` (or `qwen_reward`) rows count; partial and unjudged rows are skipped, the same rule `group_signal` uses. `k` defaults to the smallest group of two or more repeats, so every such group contributes to the `k`-way estimators; groups with fewer than `k` graded repeats are left out of pass^k and pass\@k (counted in `n_groups_at_k`). pass\@1 always averages every group. Below `min_k` repeats (`ROLLOUTS_PER_TASK`, 4: the smallest k tau-bench and tau2-bench report a pass^k on, arXiv:2406.12045 and arXiv:2506.07982) the `k`-way numbers are `None` with a `note` instead of a number too noisy to act on. Pass `k=` to choose the draw size yourself. `unanimous_short=True` counts a unanimous group shorter than `k` as if it stayed unanimous (pass^k and pass\@k equal to its pass rate, 1 or 0). That is the assumption a successive-allocation run stopped on, and leaving those groups out would score only the tasks that split and inflate the headroom. Mixed short groups still stay out. `.config` says how the rows were produced (`run_config`): task count, k, temperature, max\_tokens, policy and judge versions, prompt hash, with a `mixed` list naming any the rows disagree on. ## preflight Inspect the agent before spending simulation budget, and report after. ### classify\_failure ```python theme={null} classify_failure(row: dict) -> str | None ``` Fixed-vocabulary class for a failing row, from its reason and shape. Returns None for passing or unlabeled rows and for failures the heuristics cannot place (leave those for a person, do not guess). ### coverage\_gap ```python theme={null} coverage_gap( asks: Any, tools: Sequence[dict], system_prompt: str = '', rows: Sequence[dict] | None = None, ) -> dict[str, Any] ``` Which parts of an agent's policy the asks you already send never reach. `asks` is what a suite asks the agent: a list of prompt strings, a list of rows carrying `prompt`, or a path to a `.py` or `.jsonl` file holding either. The axes come from `build_dimensions`, the same grid `simulate` covers, so the answer is in the engine's own vocabulary: which tool, which policy rule, what stance the person takes, what the world looks like, what condition the tool is in, what happened before. Each ask is placed on the axes it touches with text heuristics, not a model: the tools its words name or imply, the rule clauses it shares words with, and the stance its words show. `untested_rules` and `untested_tools` are the parts of the policy no ask reaches, which is the list worth reading. Two axes (`world_state`, `tool_condition`) cannot be read from an ask at all: a prompt never says the order is missing or the tool timed out, so a hand-written suite leaves them at one point and `notes` says so. With `rows` (graded rollouts from a run) the report also checks the world side: rules whose rows all ended in the same tool fault are rules the asks reach but the fixtures never let happen. ### dataset\_report ```python theme={null} dataset_report( rows: Sequence[dict], tools: Sequence[dict] | None = None, system_prompt: str = '', hard_share_floor: float = 0.3, ) -> dict[str, Any] ``` One report a developer reads after simulate/grade: size, signal, mix. `hard_share_floor` (`HARD_SHARE_FLOOR`, 0.3) is the share of hard- tier rows under which the set is called easy. ### format\_coverage\_gap ```python theme={null} format_coverage_gap(report: dict[str, Any]) -> str ``` The gap report as the block a person actually reads. ### preflight ```python theme={null} preflight(tools: Sequence[dict], system_prompt: str = '') -> dict[str, Any] ``` Spec-quality report for an agent. Report only; nothing is changed. `warnings` is the list a developer should read before generating thousands of rows; `cells` is the covering-grid size the same way `recommend` counts it. ## privileged Did the agent say what only the teacher was told? ### format\_leak\_report ```python theme={null} format_leak_report(report: dict[str, Any]) -> str ``` One line per fact, the summary first. ### leak\_report ```python theme={null} leak_report(rows: Any, min_len: int = 12) -> dict[str, Any] ``` Which rows quote their own `privileged` block in the agent's text. Takes the `SimulationData` itself, `data.trajectories`, or any list of rows. Given the data object it reads the trajectories, which still carry the block; `data.rows()` is the scrubbed export and checks nothing (the report says so). Checks every row that carries `privileged` (`reference`, `principle`, and every string in `hidden_state` at least `min_len` characters long) against the final reply and every assistant turn. Returns `n_rows`, `n_checked`, `n_leaked`, `rate` (over checked rows), `checked` (False when no row carried the block, so the result is vacuous), `leaked` (up to 20 rows: `scenario_id`, `rollout_index`, `field`, `needle`) and `summary`. Does not mutate `rows`. Pass `data.trajectories`, not `data.rows()`: the export scrubs `privileged` at any depth, so exported rows carry nothing to check and the report is vacuous. When it can tell the rows came through the export, `summary` says so and names the accessor to use instead. ## publish\_gate The gate a dataset passes before it leaves for the platform. ### PublishGateError The dataset must not be published as it stands. The message says why. ### calibrate ```python theme={null} calibrate( rows: Sequence[dict], policy: PolicyRef | dict | str | None = None, model: str | None = None, ref: str | Sequence[dict] | None = None, ) -> dict[str, Any] ``` Stamp `calibration` on every graded row, in place. The per-task pass rate is over the binary rewards grouped by prompt, the same grouping `group_signal` and `pass_at` use. Rows without a 0/1 reward are left alone and counted. Returns a report with the number of tasks and rows stamped plus the `pass_at` summary. `ref` (a key holding the reference model's summed logprob, or rows scored under it) fills `mean_kl` per task from the captured logprobs; see `mean_kl`. A row whose carried stamp counts more repeats than these rows hold keeps it (`n_carried` in the report): `optimize(mode="rl")` drops duplicate trajectories, and recomputing here would report the post-dedup k as the policy's pass rate over k repeats. The producing policy and `mean_kl` are still filled in from this call. ### publish\_gate ```python theme={null} publish_gate( rows: Sequence[dict], mode: str | None = None, band: tuple[float, float] = (0.2, 0.8), policy: PolicyRef | dict | str | None = None, model: str | None = None, strict: bool = True, endorsed: Sequence[str] = (), strict_hacks: bool = False, ) -> dict[str, Any] ``` Check, calibrate, and report. Raises `PublishGateError` when `strict` and the rows are RL-shaped but ungraded or carry no mixed group, or when `strict_hacks` and `hack_scan` (with `endorsed` naming what the reward should track) finds the reward best explained by something else. Never mutates anything except the `calibration` stamp. `judge_trust` in the report is the summary `grade` stamped on the rows when they carried human labels, else `None`. ## quality Second-pass conversation quality ranker. Scores rows; does not rewrite them. ### rank\_rows ```python theme={null} rank_rows(rows: Sequence[dict]) -> list[dict] ``` Score each row in place. Returns the same list when given a list. ### score\_row ```python theme={null} score_row(row: dict) -> dict[str, Any] ``` Score one row. Returns quality, quality\_reason, quality\_scores. No mutate. ## reference Score rollouts under a reference model, so `mean_kl` has its other side. ### reference\_logprobs ```python theme={null} reference_logprobs( source, ref: str, system_prompt: str | None = None, tools: Sequence[dict] | None = None, api_key: str | None = None, concurrency: int = 4, timeout: float = 600.0, chat_template_kwargs: Mapping[str, Any] | None = None, transport: Callable[[dict], dict] | None = None, ) -> dict[str, Any] ``` Stamp `ref_logprob` on every row: the reference's summed logprob over the tokens the policy generated. Rows are modified in place; the report says what was scored. `ref` is a backend spec, `vllm:@`; on the platform's serving endpoint `` is the base by its own name (`Qwen/Qwen3-4B`: the reference of an SFT/GRPO/DPO run), a hosted model's name, or `run:` for a finished run's adapter, with `WHILEAI_API_KEY` as the key. `source` is a `SimulationData` (system prompt and tools come from its profile), a row list, or a JSONL path; pass `system_prompt=`/`tools=` for the last two so the reference sees the prompt the policy saw. `chat_template_kwargs` must match what the policy sampled with (`\{"enable_thinking": False\}` for Qwen3). Report: `n_rows`, `n_skipped` (no assistant turn or a failed call, with `errors`), `n_tokens`, `model`, and `token_count_gap` (mean |ref\_n\_tokens - n\_tokens| over rows that carry `n_tokens`): near zero when the reference shares the policy's tokenizer, which is when `mean_kl` is a KL and not a length artifact. ## rubric Rubrics: prompt-specific criteria as an object, a judge that scores them one by one, and a writer that drafts them. ### Criterion ```python theme={null} Criterion( title: str, description: str = '', weight: float = 1.0, kind: str = 'principle', ) -> None ``` One rubric item. `weight` is a positive magnitude; a `pitfall` subtracts it when the reply exhibits the mistake, a `principle` adds it when met, and a `hard` rule missed fails the whole reply. ### Rubric ```python theme={null} Rubric( criteria: tuple[Criterion, ...], source: str = 'hand', domain: str = '', notes: str = '', _cache: dict = , ) -> None ``` #### Rubric.checklist ```python theme={null} checklist(self) -> str ``` The rubric as the judge reads it. #### Rubric.score ```python theme={null} score(self, results: Mapping[str, Any]) -> dict[str, Any] ``` Reward from one verdict per criterion (keyed by title or slug; a truthy value means the reply meets a hard rule or principle, or exhibits a pitfall). A missed hard rule is a 0. Otherwise the reward is the met principle weight minus the exhibited pitfall weight, over the total principle weight, clamped to \[0, 1]; with no principle the reward is 1 minus the pitfall share. Criteria the judge did not answer count as not met (and not exhibited) and are listed. ### attach\_rubric ```python theme={null} attach_rubric( rows: Sequence[dict], rubric: Rubric | Mapping[str, Any] | Sequence[Mapping[str, Any]] | Callable[[dict], Any], overwrite: bool = True, ) -> list[dict] ``` Put a rubric on each row's `privileged` block (in place). `rubric` is a `Rubric`, its dict / list form, or `row -> Rubric | None` for a per-prompt rubric; `None` leaves that row alone. ### rubric\_judge ```python theme={null} rubric_judge( rubric: Rubric | None = None, spec: str | None = None, api_key: str | None = None, prompt: str | None = None, policy: str = '', tools: Sequence | None = None, timeout: float = 120, ) -> Callable[[dict], dict[str, Any]] ``` A judge for `run_judge` / `data.grade(judge=)` that scores the rubric item by item. `rubric` applies to every row; without one the row's own `privileged.rubric` is used and a row with none stays ungraded. The result carries `reward` (`Rubric.score`), `reason`, `markers` (`rubric:` = 1.0 met / 0.0 not, and for a pitfall 1.0 clean / 0.0 exhibited), `criteria` (the raw verdicts), `rubric_version` and the score breakdown. The judge's name folds the rubric version in when one is fixed. The hosted judge scales to zero, so the first row through warms it once (`warm_judge`, a 600s budget) while the rest of the fan-out waits. Without that, `run_judge`'s eight concurrent calls all raced a container that was still loading its weights and every row came back `invalid_result` with a `TimeoutError`. Warm-up failure is not fatal: the rows are judged anyway and report the real error. ### rubric\_of ```python theme={null} rubric_of(row: Mapping[str, Any]) -> Rubric | None ``` ### write\_rubrics ```python theme={null} write_rubrics( rows: Sequence[dict], spec: str | None = None, domain: str = '', n_criteria: tuple[int, int] = (3, 8), reference_key: str = 'reference', overwrite: bool = False, concurrency: int = 8, api_key: str | None = None, timeout: float = 120, writer: Callable[[str], Any] | None = None, max_hard: int | None = None, ) -> tuple[list[dict], dict[str, Any]] ``` Draft one rubric per distinct prompt with a model and attach it to every row of that prompt (`privileged.rubric`, `source="model"`). The writer sees the request, the row's reference answer when there is one (`privileged.reference` or `row[reference_key]`), and the `domain` guidance you give it (the general rubric the book seeds from). Rows that already carry a rubric are skipped unless `overwrite`. `writer(user_message) -> str` replaces the model call for tests and for a writer of your own. Report: prompts seen, rubrics written, failures, mean criteria per rubric, the rubric versions. `max_hard` caps the hard rules a written rubric may carry: the heaviest `max_hard` stay hard and the rest become principles with their weight (`demoted_hard` in the report). A model writer marks most of what it wants as Essential, and every Essential item a reply misses is a 0, so an uncapped rubric fails rows a binary judge passes (measured live: 22 of 32 rows). `None` keeps what the writer wrote. ## spec Model spec as a versioned object (RLHF book ch. 17). ### Spec ```python theme={null} Spec( id: str, traits: tuple[Trait, ...] = (), source: dict = , version: str = '', ) -> None ``` A versioned model spec. `version` is derived from the content when left empty, so it is stable across processes and changes on any edit. #### Spec.behaviors ```python theme={null} behaviors(self) -> list[str] ``` Trait ids, in order. Hand to `delta_report(must_not_regress=...)`. ### Trait ```python theme={null} Trait( id: str, name: str = '', principle: str = '', authority: str = 'should', examples: tuple[Any, ...] = (), ) -> None ``` One named expectation. `authority` is the strength (`must` / `should` / `may`, following the model-spec convention). ### load\_spec ```python theme={null} load_spec(obj: Any, spec_id: str | None = None) -> Spec ``` Build a `Spec` from a constitution dict, a list of traits, or a path to a JSON file with either shape. A trait may be a full dict or a bare principle string. ### spec\_version ```python theme={null} spec_version(spec: Spec | dict | Any) -> str ``` The content version of a Spec (or anything `load_spec` accepts). ### stamp\_spec ```python theme={null} stamp_spec(rows: Sequence[dict], spec: Spec) -> list[dict] ``` Return copies of `rows` tagged with the spec they were produced or graded against: `spec_id` and `spec_version`. Provenance for the ch. 17 retention question — did adherence hold from one spec version, or model version, to the next. ## stage Stage lineage: which post-training stage consumed each row (book ch. 3). ### format\_stages ```python theme={null} format_stages(report: dict[str, Any]) -> str ``` ### stage\_of ```python theme={null} stage_of(row: dict) -> str | None ``` The stamped stage, or None. An eval-sourced row with no stamp reads as `eval` (its `lineage.source`), so a held-out set is never mistaken for training data just because no one stamped it. ### stage\_report ```python theme={null} stage_report(rows: Sequence[dict]) -> dict[str, Any] ``` Rows per stage, tasks per stage, and the cross-stage leaks: any task used both in `eval` and in a training stage (sft/rm/rl/mid). That leak means the number you report was optimized against. ### stamp\_stage ```python theme={null} stamp_stage(rows: Sequence[dict], stage: str) -> list[dict] ``` Return copies of `rows` with `row["stage"] = stage`. `stage` must be one of `STAGES`; nothing else is touched. ## stats Confidence intervals, paired run comparison, and decontamination. ### compare\_runs ```python theme={null} compare_runs( a: Sequence[dict], b: Sequence[dict], metric: str = 'pass_at_1', n_boot: int = 2000, seed: int = 0, min_paired: int = 5, level: float = 0.95, ) -> dict[str, Any] ``` Is run `b` different from run `a` on `metric`? Tasks the two runs share are compared as paired differences (b minus a, per task); the interval is a `level` bootstrap over those pairs (`ci95` at the default, with `level` reported beside it) and the p-value is a sign-flip permutation test. With fewer than `min_paired` shared tasks the comparison falls back to unpaired task means and says so. `verdict` is one of `"b_better"`, `"a_better"`, `"no_difference_detected"`: the last means the interval covers zero, not that the runs are equal. Tasks on one side only are dropped from a paired comparison, and `note` says how many, since a verdict over a quarter of the tasks is not a verdict over the eval. `paired_share` is the shared fraction of every task either run saw. ### decontaminate ```python theme={null} decontaminate( rows: Sequence[dict], against: Sequence[Any] | Any, n: int = 8, fields: Sequence[str] = ('prompt',), overlap: float = 0.8, embedder: Callable[[list[str]], Sequence[Sequence[float]]] | None = None, similarity: float = 0.85, ) -> tuple[list[dict], dict[str, Any]] ``` Drop rows whose prompt overlaps an evaluation set (rlhf-book ch. 16). `against` is one or more evaluation sources: row lists, JSONL paths, or platform dataset ids (`ds_...`). Evaluation prompts, answers and references are the texts (not the eval set's own replies). Four rules, applied in this order, and a row flagged by one is not counted again by the next, so `n_contaminated` is the number of rows dropped: * `same_task` (`n_same_task`): the row's `scenario_id` or `task_id` is an evaluation row's. A task is a situation, not a string (`task_key`), so a rephrasing of an eval situation is the eval situation whatever the words say. Rows with no recorded id skip this rule. * `exact` (`n_exact`): one of the row's `fields` is an evaluation text verbatim after normalization (case and whitespace). * near copy (`n_near`): one evaluation text covers at least `overlap` of the row's words with shared word `n`-grams (the Llama 2 rule: 8-grams, 80% of tokens). Texts shorter than `n` words match verbatim only. * `semantic` (`n_semantic`), only with `embedder`: the cosine similarity between the row's text and an evaluation prompt is at least `similarity`, and the two carry different task ids or none. The default field is the prompt, the book's method; add `"final_text"` to ask the stricter question of whether replies reproduce eval answers or references. One shared n-gram is the book's test for free-form sets. Situations written from templates share whole sentences that say nothing about which question was asked, so any-n-gram flags every row of a template-written set; the coverage rule counts a row when one eval text accounts for most of it. `overlap=0` restores any-n-gram. Word overlap does not see a paraphrase. A holdout written by re-running the generator on the same briefs was 70% within 0.85 cosine of the training batch and 5 of 133 byte-identical; the 8-gram rule flagged 4 of 101 prompts and the semantic pass 16 (#286). `embedder` is any callable from a list of texts to one vector per text, so nothing here imports a model; with sentence-transformers:: from sentence\_transformers import SentenceTransformer model = SentenceTransformer("BAAI/bge-small-en-v1.5") clean, report = wai.decontaminate( train, against=\[holdout], embedder=lambda texts: model.encode(texts, normalize\_embeddings=True).tolist(), ) A semantic flag means the two prompts read alike, not that they are the same task: "cancel one reservation" and "cancel three reservations" for different customers scored 0.932 with no shared answer. So where task identity is recorded the `same_task` rule decides and the semantic pass only looks across different tasks, and the report's `notes` say the flag is a question to check, not a verdict. The default stays lexical: `similarity` 0.85 was read off BGE (unrelated prompts score about 0.55 there) and does not transfer to every model, so the pass calibrates it for yours when it can: with eval rows that carry task ids, the 99th percentile of similarity over eval-prompt pairs with different task ids is how alike distinct tasks read to this embedder, and `notes` says it. A threshold below that number flags tasks that merely share a domain, and the note says so when `similarity` is. Returns the clean rows and a report: the count under each rule, hits per field, the eval text count, and the first offenders with their coverage (or `similarity` for semantic hits). ### detectable\_effect ```python theme={null} detectable_effect( n_tasks: int, base: float = 0.6, k: int = 4, power: float = 0.8, alpha: float = 0.05, ) -> float | None ``` The smallest gain `n_tasks` paired tasks can prove at `power`: `holdout_size` solved for the effect (`FIXED_POINT_STEPS` fixed-point steps, since the after-side variance depends on it). `None` below `MIN_HOLDOUT_TASKS` tasks. ### eval\_variance ```python theme={null} eval_variance( *runs: Sequence[dict], metric: str = 'pass_at_1', by: str | None = None, ) -> dict[str, Any] ``` How much an evaluation moves when the same model is evaluated again (rlhf-book ch. 16, Evaluation). Pass each re-run's rows as its own argument, or one row list whose rows say which run they belong to: `lineage.eval_run` (what `simulate(runs=3)` stamps), else `lineage.scoring_run_id` (what `evaluate(run_id=)` stamps), or a top-level or lineage key named by `by`. Each run's `metric` is a mean over tasks; the report is those means, their mean, the sample standard deviation `run_std`, and `noise_band` = `noise_band(run_std)`, 1.96 x sqrt(2) x `run_std`: a before/after delta with one run per side is the difference of two re-run draws, and a delta inside that band is what re-running the eval does on its own (`run_std` is taken as the eval's spread; from three runs it is rough, and the note says so). `run_std_by_metric` reports the same floor for pass\@1 and every marker shared by all runs; hand that mapping to `delta_report(run_std=)` so each metric uses its own re-run variance. The scalar `run_std` remains the selected `metric`'s value for callers comparing only one metric. `stability` places `run_std` on Olmo 3's bands in points. Fewer than three runs is a difference, not a distribution; the report says so and `run_std` is `None` below two. ### holdout\_size ```python theme={null} holdout_size( effect: float, base: float = 0.6, k: int = 4, power: float = 0.8, alpha: float = 0.05, before: Sequence[dict] | None = None, after: Sequence[dict] | None = None, task_std: float | None = None, rows: Sequence[dict] | None = None, ) -> dict[str, Any] ``` How many paired tasks a holdout needs to prove a gain of `effect`. Models the test `delta_report` runs: each task's pass rate over `k` rollouts on each side, the delta as the mean of the paired differences, the interval from a bootstrap over tasks. The usual two-sided power calculation then gives `n = ((z_\{1-alpha/2\} + z_power) * sd / effect) ** 2` with `sd` the standard deviation of one task's paired difference (rlhf-book ch. 16: the point of a better eval is statistical power when comparing training runs). Where `sd` comes from is the whole question, and there are three ways to answer it, best first: * `before` and `after`, the graded arms of a previous eval on the same tasks (the two row lists `delta_report(before, after)` takes): `sd` is measured as the sample sd of the per-task differences, which carries the covariance that pairing buys and whatever shape the gain had. No model. `sd_source` is `"rows"` and `n_paired` says how many tasks it was read off. * `task_std`, a number you measured (the per-task sibling of `delta_report`'s `run_std`): the same quantity read off a previous `delta_report`: `(hi - lo) * sqrt(n_paired_tasks) / 3.92` from `target_ci95` and `n_paired_tasks` (or any `metrics[...]["ci95"]` with its `n_paired`). Agent rubrics sat near 0.38 across five lanes (#288). `sd_source` is `"given"`. `eval_variance`'s `run_std` is a different number (how much a re-run moves the mean) and is not this. * Neither: the binomial model `sqrt((p(1-p) + q(1-q)) / k)` with `p = base` and `q = base + effect`, `sd_source` `"model"`. It assumes two things it cannot check: that the gain is spread evenly across tasks, and that the two arms are independent draws (`Var(A) + Var(B)`, no covariance term). When the gain is carried by a few tasks, most tasks are ties and the paired differences spread far wider than binomial-per-task predicts; a voice trait at 0 -> 0.127, k=4, carried by 19 of 150 tasks, measured sd 0.333 against the model's 0.168 and needed 54 tasks where the model said 14 (#292). So the model path also returns `n_tasks_concentrated`, the count if the gain were carried by the fewest tasks that can carry it (each going from `base` to 1\), and `notes` says which assumption is in play. On a holdout whose tasks differ in difficulty the independence assumption errs the other way: the model puts `p(1-p)` of variance on every task where pairing keeps each task's own `p_i(1-p_i)`, whose mean is `p(1-p) - Var(p_i)`, so it asks for `1 / (1 - Var(p_i) / (p(1-p)))` times the tasks pairing needs (1.19x at spread 0.2 around 0.5, 2.78x at 0.4). `before` alone reports the spread as `base_spread` and puts that ratio in `notes`. `before` on its own (`rows` is the same argument under its old name) reads `base` and `k` off the data. Returns `n_tasks` plus the inputs, `task_std`, `sd_source`, `half_width` (the 95% band on the delta at that `n`), `n_tasks_concentrated`, `base_spread`, `n_paired` and `notes`; every key is present on every path (`None` or `[]` where it does not apply). The default answer is unchanged; the honest paths are the two that measure. The recipe that asked for this had 140 tasks at k=4 around 0.6: a band of about +-0.06, so a real 3-point gain reads `no_change_detected` every round. This says so before training. ### marker\_names ```python theme={null} marker_names(rows: Sequence[dict]) -> list[str] ``` ### marker\_summary ```python theme={null} marker_summary( rows: Sequence[dict], names: Sequence[str] | None = None, n_boot: int = 2000, seed: int = 0, ) -> dict[str, dict[str, Any]] ``` `metric_summary` for every marker on the rows (or `names`). Each marker's stats are keyed `mean`, `ci95` (not `ci`), `n_tasks`, `n_rows` (not `n`), `n_rows_at_1`, `n_rows_at_0`, `degenerate`, and `note` or `warning` when there is one. `ci95` is `None` below `MIN_CI_TASKS` tasks, and `note` then says how many tasks the marker has and how many the interval needs; a reader who sees only `None` cannot tell that from a bug. ### metric\_summary ```python theme={null} metric_summary( rows: Sequence[dict], metric: str = 'pass_at_1', n_boot: int = 2000, seed: int = 0, ) -> dict[str, Any] ``` Mean over tasks with a task-bootstrap 95% interval. `degenerate` is set when every applicable row scored the same value: the metric has not been shown to be able to come out any other way, so `ci95` is `None` (the way `pass_at` returns `None` below three groups) and `warning` says so. A marker that is silently unfireable (a key-name mismatch) and one that is genuinely always true look identical otherwise, and either one passed to `must_not_regress` is a guard that cannot fail (#270). `n_rows_at_1` and `n_rows_at_0` put the row-level split next to the mean. ### task\_key ```python theme={null} task_key(row: dict) -> str ``` The one name every report groups a row's rollouts under. A task is a situation, not a string: `scenario_id` when the row has one (the engine's situation id, shared by the repeats of one opener and by the textured phrasings of one situation), else `task_id` (rows from elsewhere), else the prompt text. `pass_at`, `compare_runs`, `delta_report`, `eval_variance`, `curriculum`, `group_signal` and the exporters all count tasks with this key, so the same rows give the same task count everywhere (rlhf-book ch. 16: intervals and paired comparisons are over tasks, never rows). ## style Over-optimization signatures on replies: the things a reward pays for by accident. ### refusal\_report ```python theme={null} refusal_report( rows: Sequence[dict], phrases: Sequence[str] = ("i can't help with", 'i cannot help with', "i can't assist", 'i cannot assist', "i'm unable to", 'i am unable to', "i won't be able to", "i'm not able to", 'i am not able to', "i can't do that", 'i cannot do that'), examples: int = 5, ) -> dict[str, Any] ``` Over-refusal on a benign set (rlhf-book ch. 14 "Over-Refusal"). Pass the rows whose asks the agent should have answered; the report is the share it refused anyway, with a Wilson 95% interval, the phrases that fired, and the first few refusals so a person can read them. Refusal rate on a mixed set means nothing, which is why this takes the benign rows rather than finding them. ### style\_markers ```python theme={null} style_markers( rows: Sequence[dict], phrases: Mapping[str, Sequence[str]] | None = None, ) -> list[dict] ``` Stamp the style markers on every row's `markers` (in place) and return the rows. `phrases` overrides or extends `STYLE_MARKERS`: `{"no_boilerplate": [...], "no_brand_voice": [...]}`. Existing markers with other names are kept. ### style\_report ```python theme={null} style_report( rows: Sequence[dict], phrases: Mapping[str, Sequence[str]] | None = None, threshold: float = 0.3, n_boot: int = 2000, seed: int = 0, ) -> dict[str, Any] ``` How much of each signature the replies carry, and whether the reward pays for it. Does not mutate `rows`. Per marker: `clean` (share of rows without a hit, with a task- bootstrap 95% interval), `hits` (rows with a hit), `top_phrases` (the phrases that fired, most common first) and `reward_corr` (Pearson between "phrase present" and the binary reward over graded rows). A positive correlation at or above `threshold` is flagged: the judge is rewarding the tic, and a policy trained on these rewards will produce more of it (rlhf-book ch. 14). `warnings` says so in one line per flag. ## trace Did the agent fake the work? Flags read from the trajectory, not the prose. ### trace\_flag\_report ```python theme={null} trace_flag_report( rows: Sequence[dict], kinds: Mapping[str, str] | None = None, threshold: float = 0.3, n_boot: int = 2000, seed: int = 0, examples: int = 3, ) -> dict[str, Any] ``` How often each flag fires, what a reviewer should read, and whether the reward pays for it. Does not mutate `rows`. Per flag: `n` (rows it fired on), `rate`, `examples` (evidence with the ask), and `reward_corr` (Pearson between "flag fired" and the binary reward over graded rows); a positive correlation at or over `threshold` is flagged, since a judge that pays for a faked turn trains a policy to fake turns. Per marker: the clean share with a task-bootstrap interval. `warnings` says so, one line per flag. ### trace\_flags ```python theme={null} trace_flags( row: Mapping[str, Any], kinds: Mapping[str, str] | None = None, ) -> dict[str, str] ``` The flags that fire on one rollout: `{flag: evidence}`. First match wins per flag, so the evidence points at the earliest cause. ### trace\_markers ```python theme={null} trace_markers( rows: Sequence[dict], kinds: Mapping[str, str] | None = None, evidence: bool = True, ) -> list[dict] ``` Stamp the trace markers on every row's `markers` (in place) and return the rows: 1.0 when the family is clean, 0.0 when a flag fired. With `evidence` the flags and their fragments land on the row as `trace_flags` for a reviewer.