whileai.simulations.score

Grading, judge checks, pass@k, selection for SFT and RL, hack scans. 99 public names. import whileai.simulations as wai, then wai.name.

Name	What it does
`Criterion`	One rubric item.
`PassAt`	pass@1 / pass^k / pass@k over graded groups.
`PublishGateError`	The dataset must not be published as it stands.
`Rubric`
`ScoredData`	Scored trajectories: the one representation grade and eval share.
`Spec`	A versioned model spec.
`Trait`	One named expectation.
`annotator_agreement`	How the annotators on `gold_labels` agree with each other.
`argument_grounding`	1.0 when every string argument of every tool call is grounded in the conversation (a rollout with no calls is grounded), else 0.0.
`attach_labels`	Write hand labels onto rows (in place) and return `(rows, report)`.
`attach_rubric`	Put a rubric on each row’s `privileged` block (in place).
`audit_grades`	Estimate the verifier’s false-negative rate from a judged sample.
`behavior_signature`	Hash of tool sequence, argument provenance, statuses, and outcome shape.
`behavioral_markers`	Rate of each stock marker over `rows` (fraction of rollouts that trip it).
`build_preference_pairs`	Same-task chosen/rejected pairs for preference training (DPO-style).
`calibrate`	Stamp `calibration` on every graded row, in place.
`classify_failure`	Fixed-vocabulary class for a failing row, from its reason and shape.
`compare_runs`	Is run `b` different from run `a` on `metric`?
`conduct_grade`	Score agent conduct.
`coverage_gap`	Which parts of an agent’s policy the asks you already send never reach.
`curriculum`	Split graded tasks into a training curriculum by measured difficulty.
`dataset_report`	One report a developer reads after simulate/grade: size, signal, mix.
`decontaminate`	Drop rows whose prompt overlaps an evaluation set (rlhf-book ch.
`dedupe_groups`	Drop repeat trajectories within one ask.
`delta_report`	Compare `after` to `before` on pass@1 and every shared marker.
`detectable_effect`	The smallest gain `n_tasks` paired tasks can prove at `power`: `holdout_size` solved for the effect (`FIXED_POINT_STEPS` fixed-point steps, since the after-side variance depends on it).
`eval_variance`	How much an evaluation moves when the same model is evaluated again (rlhf-book ch.
`evaluate`	Judge held-out rollouts under the exact contract `grade` uses.
`expected_outcome`	What the checklist will look for on this task, in one sentence.
`filter_rl_rows`	Split keep/drop.
`format_audit`	The block a person reads: the summary, then the reasons.
`format_coverage_gap`	The gap report as the block a person actually reads.
`format_curriculum`	One-line-per-fact summary for a terminal.
`format_delta_report`	The block a person reads: headline, then one line per metric.
`format_hack_scan`	The block a person reads: the regime, the floor, the ranking.
`format_hack_scan_diff`	The block a person reads: what was learned, then the shifts.
`format_judge_trust`
`format_leak_report`	One line per fact, the summary first.
`format_markers`	One line per marker, highest rate first.
`format_stages`
`grounding_report`	Over a row set: the share of rollouts with every argument grounded, the share with any call at all, and the most common invented values by tool and key, for a reviewer to look at.
`group_signal`	Within-ask contrast.
`hack_scan`	Rank what separates reward within each ask against a permutation noise floor, and say what a grouped update would learn.
`hack_scan_diff`	What the policy learned: the scan before training against the scan after, on rollouts scored by the same reward.
`holdout_size`	How many paired tasks a holdout needs to prove a gain of `effect`.
`judge_agreement`	Agreement between the judge’s `reward` and a trusted label.
`judge_pairs`	Ask a judge which side of each pair is better, both ways round.
`judge_probes`	Try the reward hacks a policy finds first on the judge, on purpose.
`judge_trust`	The judge-trust report.
`leak_report`	Which rows quote their own `privileged` block in the agent’s text.
`length_report`	Truncated rollouts and asks whose reply lengths are far apart.
`load_spec`	Build a `Spec` from a constitution dict, a list of traits, or a path to a JSON file with either shape.
`logprob_report`	Coverage and shape of the captured logprobs.
`mark_grounding`	Copies of `rows` with `markers["argument_grounding"]` stamped, so `marker_summary`, `delta_report` and the run page read it.
`mark_rows`	Return copies of `rows` with the stock markers merged into each row’s `markers` dict, ready for `marker_summary` / `delta_report`.
`marker_names`
`marker_summary`	`metric_summary` for every marker on the rows (or `names`).
`mean_kl`	Sampled `KL(pi \|\| pi_ref)` per generated token, overall and per task.
`metric_summary`	Mean over tasks with a task-bootstrap 95% interval.
`near_duplicate_prompts`	Pairs of distinct asks whose token sets overlap at or above `threshold`.
`next_round`	The prompt set for the next round, from the last round’s graded rollouts.
`normalize_judge_result`	Coerce one judge return into the contract; never invent a reward.
`optimize`	One call after grading: concentrate for the post-training target.
`outcome_check`	(outcome, reason, checks).
`pairwise_judge`	A model judge for `judge_pairs`: `judge(a_row, b_row) -> \{"winner": "A" \| "B" \| "tie" \| None, "reason": str\}`.
`pass_at`	pass@1, pass^k and pass@k from graded rows, grouped by task.
`preflight`	Spec-quality report for an agent.
`privileged_context`	The teacher’s block for a row at birth: `hidden_state` (what the world knows that the ask does not say) and `reference` (what the checklist expects).
`publish_gate`	Check, calibrate, and report.
`rank_rows`	Score each row in place.
`recommend`	How much data this agent needs, from its own grid.
`reference_logprobs`	Stamp `ref_logprob` on every row: the reference’s summed logprob over the tokens the policy generated.
`refusal_report`	Over-refusal on a benign set (rlhf-book ch.
`retire_solved`	Return the rows with every solved task removed.
`reward_correlations`	corr(reward, feature) for the cheap features a judge tends to reward by accident: reply length, tool-call count, assistant turns, and the over-optimization signatures of rlhf-book ch.
`row_markers`	The stock markers for one row’s final text.
`rubric_judge`	A judge for `run_judge` / `data.grade(judge=)` that scores the rubric item by item.
`rubric_of`
`run_judge`	Score trajectories with any judge.
`score_row`	Score one row.
`select_for_rl`	Whole mixed groups up to roughly `target` rows.
`select_for_sft`	Diverse correct demonstrations, at most `target` rows.
`spec_version`	The content version of a Spec (or anything `load_spec` accepts).
`stage_of`	The stamped stage, or None.
`stage_report`	Rows per stage, tasks per stage, and the cross-stage leaks: any task used both in `eval` and in a training stage (sft/rm/rl/mid).
`staleness_report`	Which policies produced these rows, and can an update still use them.
`stamp_spec`	Return copies of `rows` tagged with the spec they were produced or graded against: `spec_id` and `spec_version`.
`stamp_stage`	Return copies of `rows` with `row["stage"] = stage`.
`style_markers`	Stamp the style markers on every row’s `markers` (in place) and return the rows.
`style_report`	How much of each signature the replies carry, and whether the reward pays for it.
`task_checklist`	Judge contract: conduct gated by the task’s checkable outcome.
`task_key`	The one name every report groups a row’s rollouts under.
`trace_flag_report`	How often each flag fires, what a reviewer should read, and whether the reward pays for it.
`trace_flags`	The flags that fire on one rollout: `{flag: evidence}`.
`trace_markers`	Stamp the trace markers on every row’s `markers` (in place) and return the rows: 1.0 when the family is clean, 0.0 when a flag fired.
`trim_out_of_band`	Difficulty band filter.
`trim_unanimous_groups`	Drop asks whose k >= `min_k` rollouts all landed 0 or all landed 1.
`ungrounded_arguments`	The string arguments of the rollout’s tool calls that appear nowhere in the context the call could draw on.
`write_rubrics`	Draft one rubric per distinct prompt with a model and attach it to every row of that prompt (`privileged.rubric`, `source="model"`).

agreement

Does the judge agree with labels you trust?

judge_agreement

judge_agreement(
    rows: Sequence[dict],
    gold: str | Sequence[dict] = 'gold_reward',
    reward: str = 'reward',
    allow_model_gold: bool = False,
) -> dict[str, Any]

Agreement between the judge’s reward and a trusted label. gold is either a key on the same rows (default gold_reward, the field attach_labels fills when you hand-label a sample) or a second row list from another scoring pass, matched by rollout id, scenario id plus rollout index, or prompt plus final text. Only exact 0/1 labels on both sides count; partial scores and unjudged rows are reported as skipped, not guessed. Returns n, agreement, kappa (Cohen, chance-corrected), the confusion counts, pass_when_gold_fail (the leak rate: gold failures the judge passed) and fail_when_gold_pass, both pass rates, gold_kind (where the labels came from: "human", "model", "unknown" for rows with no record), ok (rows were compared and the labels are a person’s), and warnings. A second judge pass is model gold; rows with gold_reward but no gold_kind are unknown; either makes ok false with the reason unless allow_model_gold=True.

audit

Is the verifier failing answers that are right?

audit_grades

audit_grades(
    rows: Sequence[dict],
    judge: Callable[[dict], Any],
    sample: int = 40,
    passes: int = 0,
    question: str = "The reference answer is the answer key for this task. Is the reply's final answer correct, meaning it gives the same result as the reference? Differences in formatting, rounding, ordering (unless the task asks for an order), column or field naming, or an equivalent way of expressing the same result do not make it wrong. A missing, different, or partial result does.",
    seed: int = 0,
    concurrency: int = 4,
    timeout: float = 120,
    judge_name: str | None = None,
    fn_warn: float = 0.1,
) -> dict[str, Any]

Estimate the verifier’s false-negative rate from a judged sample. rows are graded by the verifier (run_judge(rows, verifier) or data.grade(judge=verifier)): reward 0/1, reason from the rule. sample failed rows (reward 0, judge ok) are drawn with seed and each is put to judge (any judge in the run_judge contract: rubric_judge(), grade_llm, your own callable) with the reference in place and question as the rubric when the row carries none. A judge reward at or above PASS_THRESHOLD (0.5) on a failed row is a false negative. passes samples passed rows the same way for the false-positive side. Returns fn_rate with fn_ci95 (Wilson), estimated_wrong_fails (the rate over every failed row), reasons (the verifier’s failure kinds in the sample, each with how many the judge overturned), a few examples, fp_rate when passes > 0, and warnings. Above fn_warn (FN_WARN, 0.10) the summary says to fix the verifier before training; select_for_rl(audit=report) and optimize(audit=) carry the same warning into the selection.

format_audit

format_audit(report: dict[str, Any]) -> str

The block a person reads: the summary, then the reasons.

checklist

A task-specific checklist reward, derived from what the world knows.

expected_outcome

expected_outcome(dims: dict | None) -> str | None

What the checklist will look for on this task, in one sentence. The same branches as outcome_check, read before the agent has acted. None when no rule applies (a compound ask). This is the teacher’s privileged.reference: the answer key the student must never be shown.

outcome_check

outcome_check(row: dict) -> tuple[int | None, str, dict[str, Any]]

(outcome, reason, checks). None when no rule applies to this task.

privileged_context

privileged_context(dims: dict | None, faults: dict | None = None) -> dict

The teacher’s block for a row at birth: hidden_state (what the world knows that the ask does not say) and reference (what the checklist expects). Empty when the task carries neither.

task_checklist

task_checklist(
    row: dict,
    declared_tools: set[str] | None = None,
) -> dict[str, Any]

Judge contract: conduct gated by the task’s checkable outcome.

curriculum

Curriculum: order tasks easy-to-hard and retire the solved ones.

curriculum

curriculum(
    rows: Sequence[dict],
    solved: float = 0.8,
    floor: float = 0.2,
    band: tuple[float, float] = (0.2, 0.8),
    tiers: int = 3,
    min_rollouts: int = 2,
) -> dict[str, Any]

Split graded tasks into a training curriculum by measured difficulty. A task is solved when its pass rate is above solved (retire it: an all-pass task is dead gradient). It is not ready when its pass rate is below floor (hold it: no signal until the policy can sometimes solve it). Everything from floor to solved inclusive is trainable, ordered easy to hard (highest pass rate first) and split into tiers difficulty buckets for a staged schedule. The defaults are the two edges of DEFAULT_BAND (20% and 80%), the same band select_for_rl keeps, so a task at 1 of 8 is not ready here and out of band there for the same reason. Tasks with fewer than min_rollouts graded rollouts cannot have a difficulty and are reported separately. Returns a report; nothing is mutated. band is recorded and used only to count how many trainable tasks sit in the reasoning-recipe 20-80% sweet spot, so you can see whether the set has usable signal.

format_curriculum

format_curriculum(report: dict[str, Any]) -> str

One-line-per-fact summary for a terminal.

retire_solved

retire_solved(
    rows: Sequence[dict],
    solved: float = 0.8,
    min_rollouts: int = 2,
) -> list[dict]

Return the rows with every solved task removed. A task above the solved pass rate teaches nothing, so its rollouts are dropped; tasks with too few rollouts to judge are kept.

delta

Did training move the behavior, and did anything else slip?

delta_report

delta_report(
    before: Sequence[dict],
    after: Sequence[dict],
    target: str | None = None,
    must_not_regress: Sequence[str] = (),
    markers: Sequence[str] | None = None,
    by: str | Callable[[dict], Any] | None = None,
    run_std: float | Mapping[str, float | None] | None = None,
    run_std_runs: int | None = None,
    proxy: str | None = None,
    n_boot: int = 2000,
    seed: int = 0,
    balance_rollouts: bool = False,
    alpha: float = 0.05,
    power: float = 0.8,
    ceiling_pass_rate: float = 0.9,
    answered_gap_points: float = 0.1,
    answered_alpha: float = 0.01,
) -> dict[str, Any]

Compare after to before on pass@1 and every shared marker. alpha is the false-positive rate every verdict runs at: each interval is at 1 - alpha (ci95 at the default), the re-run band uses the same quantile, and family_error is 1 - (1 - alpha) ** n_metrics. power feeds the sizing line (detectable_effect, holdout_size). ceiling_pass_rate, answered_gap_points and answered_alpha are the flags’ thresholds (CEILING_PASS_RATE, ANSWERED_GAP_POINTS, ANSWERED_P_MAX). The two sides should have the same number of rollouts per task. When a run lost rollouts (data.report()["rollouts_lost"]), one arm can sit at k=4 and the other at k=2; the report warns, next to the sizing line, naming both. Unequal k is a precision issue, not a bias: a task’s pass rate is its mean over however many rows it has, so rows lost at random leave the paired delta unbiased and only widen its interval (simulated, k=4 against k=2 on half the tasks: mean delta on the true value, interval about 10% wider). Rows lost for a reason are the problem: a timeout that takes the hard runs, an empty reply on the long ones, and the surviving rows on that arm score higher than the arm does. No trimming fixes that; only re-running the short arm on its short tasks does, and data.report()["rollouts_lost_by"] says why the rows went missing. balance_rollouts=True (off by default) trims every paired task to the rows both sides have, chosen by seed, so pass^k and pass@k share one k; it costs precision (another 10% on the interval in the same simulation) and removes no bias (failures dropped on one arm: delta 0.32 untrimmed, 0.32 trimmed, true 0.05), and balanced says how many rows each side gave up. target names the metric the run was meant to move ("pass_at_1" or "marker:<name>"); the verdict on it is the headline. proxy names the metric the run was actually trained on (the training reward as a marker, e.g. "marker:first_action"). When the proxy moved up and the target did not, or the proxy’s interval sits entirely above the target’s, the report is over_optimized and fails: the policy learned something the target does not credit (rlhf-book ch. 14). must_not_regress lists metrics whose significant drop fails the report. Metric names for markers are the marker names; pass@1 is "pass_at_1". Tasks on one side only do not pair; their count is n_unpaired_tasks and, when any were dropped, a warning says so. by splits the target by a group on each row: a row key (top level, or a marker name) or a callable row -> group. The report gains groups: the target compared within each group, so a headline that moved cannot hide a kind of prompt that moved the other way. A group whose target dropped significantly is listed in groups_down and warned about; it does not flip ok, which stays the must_not_regress contract (name the group’s metric there if it should). run_std is the evaluation’s own re-run standard deviation (rlhf-book ch. 16, appendix C). Pass eval_variance(...)["run_std_by_metric"] so pass@1 and each marker are judged against their own floor: a marker on a subset of tasks is several times noisier than pass@1, and pass@1’s floor reads a re-run draw of it as a regression (#300). A scalar applies one floor to every metric, as before. A metric the mapping lacks, or carries as None, is never given another metric’s floor: it gets noise_note: "no_replicate_floor", a warning, and its verdict rests on the task interval alone. A metric whose delta is inside noise_band(floor, n_a, n_b, df) is within_noise: not improved, not slipped, not a regression, and a target there reads within_eval_noise rather than moved, because re-running the eval moves it that much on its own. The band is floor * sqrt(1/n_a + 1/n_b) (the delta is a mean of n_a runs against a mean of n_b) times 1.96 for a given floor, which is taken as the eval’s spread. A floor that came from re-runs is an estimate, not the spread: pass run_std_runs= (how many re-runs it was computed from, eval_variance(...)["n_runs"]) and the band uses the two-sided t quantile at df = run_std_runs - 1 instead (three re-runs: 4.30 x floor x sqrt(2) with one run per side, not 1.96; under pure noise the 1.96 band lets about one delta in five through at df=2). A given run_std without run_std_runs keeps 1.96 and a warning names the fix. When both row sets carry two or more lineage.eval_run values (simulate(tasks=..., runs=3)) the report computes each metric’s floor itself, pooled over the two sides, and uses the t quantile at df = sum(runs - 1) instead (three runs per side: 2.78 x floor x sqrt(2/3)); run_std is the headline metric’s floor, run_std_by_metric has them all, noise_band is the headline band, noise_rule spells it out, and eval_runs says how many runs each side had. With one run on either side and no run_std a target that moved reads moved_unreplicated and a warning says how to fix it: one evaluation is a draw, not a distribution (rlhf-book ch. 16, “why many comparisons are unreliable”, and appendix C). not_comparable lists every reason the two arms cannot be compared at all (none are raised here; the comparability checks add theirs). ceiling is set when the before side already passes CEILING_PASS_RATE of its tasks, or when fewer than CEILING_MIN_TASKS_WITH_ROOM paired tasks (and under half) are not already passed every time: there is little room left for an improvement to show, whatever the training did. config says what each side was produced with (pass_at(...).config per side: task count, k, temperature, max_tokens, policy and judge versions, prompt hash). A warning names each setting the two sides disagree on, and says so when both sides are the same policy version (rlhf-book ch. 16: a comparison is only as good as the settings it was run under). config[side]["answered_share"] is the share of rows per side with a spoken reply once <think> markup is gone. Every rate is conditional on it. The two shares are compared with a pooled two-proportion z test; when it clears ANSWERED_P_MAX (p < 0.01) the warning states p and the gap, and when the gap also exceeds the re-run band (or ANSWERED_GAP_POINTS with no band) the report fails with answered in not_comparable and names the mechanism: a reasoning base against a reasoning-suppressed adapter under one shared max_tokens runs out of budget inside <think> and never answers, so the adapter wins every row the base did not reply to (#297). not_comparable lists every such cause under one prefix, NOT COMPARABLE:. situations is the cause when fewer than half the tasks are on both sides (paired_share under 0.5 with tasks on one side only): the arms drew different situation sets, so the delta over the few that pair is between two evals, and the fix is to pin the after side to the before run’s tasks (tasks=) or compare per tier with dataset_report. A replay (simulate(tasks=...) or runs=N) keeps the writer of the run it replays on writer_model, so two runs of one call compare as one writer. Situations nobody’s model wrote (a seeds= ask, the offline template writer, or a replay of either) count as one writer for this check: nothing there could have moved with the weights. headline_verdict is the verdict format_delta_report prints on its first line, in words: PASS only for a gain the report supports, NO DIFFERENCE for an interval over zero, NOT COMPARABLE (causes) when the arms cannot be compared, FAIL for a regression, a guard, or over-optimization. ok is the gate (no regression, no failed guard, comparable arms); it does not say the change helped.

format_delta_report

format_delta_report(report: dict[str, Any]) -> str

The block a person reads: headline, then one line per metric.

grading

Deterministic conduct rules for any tool-using agent.

behavior_signature

behavior_signature(trajectory: dict) -> str

Hash of tool sequence, argument provenance, statuses, and outcome shape.

conduct_grade

conduct_grade(trajectory: dict, declared_tools: set[str] | None = None) -> dict

Score agent conduct. Tool/sandbox faults are a flag, not a zero.

grounding

Argument grounding: did every tool argument come from the conversation?

argument_grounding

argument_grounding(row: dict, **kwargs: Any) -> float

1.0 when every string argument of every tool call is grounded in the conversation (a rollout with no calls is grounded), else 0.0. Keyword arguments are those of ungrounded_arguments.

grounding_report

grounding_report(rows: Sequence[dict], **kwargs: Any) -> dict[str, Any]

Over a row set: the share of rollouts with every argument grounded, the share with any call at all, and the most common invented values by tool and key, for a reviewer to look at.

mark_grounding

mark_grounding(rows: Sequence[dict], **kwargs: Any) -> list[dict]

Copies of rows with markers["argument_grounding"] stamped, so marker_summary, delta_report and the run page read it.

ungrounded_arguments

ungrounded_arguments(
    row: dict,
    ignore_keys: Sequence[str] = (),
    allow: Sequence[str] = (),
    min_len: int = 3,
) -> list[dict[str, str]]

The string arguments of the rollout’s tool calls that appear nowhere in the context the call could draw on. Each entry is \{"tool", "key", "value"\}. ignore_keys skips argument names that are free text by design (a note, a message body); allow lists values that are legal without appearing in the conversation (an enum, a default); strings shorter than min_len are skipped.

hack_scan

What will the policy learn from this reward? Name it before training.

format_hack_scan

format_hack_scan(report: dict[str, Any], top: int = 12) -> str

The block a person reads: the regime, the floor, the ranking.

format_hack_scan_diff

format_hack_scan_diff(report: dict[str, Any]) -> str

The block a person reads: what was learned, then the shifts.

hack_scan

hack_scan(
    rows: Sequence[dict],
    endorsed: Sequence[str] = (),
    features: Mapping[str, Callable[[dict], float | None]] | None = None,
    auto: bool = True,
    top_k: int = 200,
    n_perm: int = 100,
    min_obs: int = 20,
    seeds: int = 12,
    reward: str = 'reward',
    seed: int = 0,
    top_features: int | None = 20,
    alpha: float = 0.05,
) -> dict[str, Any]

Rank what separates reward within each ask against a permutation noise floor, and say what a grouped update would learn. alpha (ALPHA, 0.05) sets the floor: tau is the 1 - alpha quantile of the strongest feature’s |rho| when reward is shuffled within ask, so a feature above it clears chance at that rate. rows are graded rollouts, several per ask (mode="rl"); the reward under reward may be 0/1 or partial credit. endorsed names the features the reward is supposed to track, as substrings of feature names ("lookup_order" matches tool:lookup_order and contains:lookup_order; "marker:grounded" a marker). Without it the scan still ranks and floors, but cannot call a hack a hack. features adds hand-tier columns: {"name": lambda row: value}. top_features caps the ranking in the report (None lists all). Returns regime (train, reward_hack, pool_exhausted, no_signal, degenerate, unknown), tau (the floor), features ranked by |within-ask correlation| with the pooled correlation beside each, top_feature, endorsed_on_top, integrity (share of the above-floor signal that sits on an endorsed feature), the support numbers (asks all-pass, all-fail, mixed, gradient capacity), and warnings in one line each. degenerate is the refusal: an ask holds fewer than MIN_DISTINCT_PER_ASK distinct rollouts at the median (distinct_per_ask) and two or more features sit at |rho| >= DEGENERATE_RHO, exactly collinear with reward and with each other because nothing else could happen at that variety. The ranking cannot separate them and the noise floor is no help (it tells signal from noise, not one perfect explanation from another), so top_feature and integrity are None, inverted is empty, no hack is claimed, and collinear lists the tied features. The direction is withheld with the name: at that variety an endorsed feature is negative exactly when it fell on the failing trajectory, so the sign is the same coin flip. Collinear features on a varied pool are left alone: there the ranking found two names for one behavior, and a genuinely inverted endorsed feature is still reported.

hack_scan_diff

hack_scan_diff(
    before: Sequence[dict],
    after: Sequence[dict],
    endorsed: Sequence[str] = (),
    top: int = 10,
    **scan_kwargs: Any,
) -> dict[str, Any]

What the policy learned: the scan before training against the scan after, on rollouts scored by the same reward. A feature that clears the floor after and did not before is what the update moved toward; one that dropped out is what it moved away from. gained and lost list them with both correlations, moved the largest shifts either way, and learned is the one line to read: the top gained feature, and whether it is endorsed. scan_kwargs reach both hack_scan calls. When either side comes back degenerate, every feature there is above the floor at |rho| 1 and no feature can be said to have gained it. learned says which side could not be read and why, and no hack is claimed; the rows are still listed so the shift is visible.

hygiene

Row hygiene a grouped RL update or a rejection-sampling pass cares about.

dedupe_groups

dedupe_groups(rows: Sequence[dict]) -> tuple[list[dict], dict[str, Any]]

Drop repeat trajectories within one ask. Keeps the first of each (behavior signature, normalized reply) pair. Reports how many pairs disagreed on reward, which is label noise the judge introduced.

length_report

length_report(rows: Sequence[dict], max_spread: float = 4.0) -> dict[str, Any]

Truncated rollouts and asks whose reply lengths are far apart. max_spread is max / median within one ask.

near_duplicate_prompts

near_duplicate_prompts(
    rows: Sequence[dict],
    threshold: float = 0.8,
) -> dict[str, Any]

Pairs of distinct asks whose token sets overlap at or above threshold. Quadratic in the number of asks; fine for the few thousand a run produces. Report only.

reward_correlations

reward_correlations(
    rows: Sequence[dict],
    threshold: float = 0.3,
) -> dict[str, Any]

corr(reward, feature) for the cheap features a judge tends to reward by accident: reply length, tool-call count, assistant turns, and the over-optimization signatures of rlhf-book ch. 14 (boilerplate, hedging, sycophancy, refusal phrases, 1 when present; see score.style), plus every trajectory flag that fired on any row (lie.*, hack.*, risk.*; see score.trace). Any |corr| at or above threshold is flagged. A negative tool-count correlation means the reward pays the policy to do less; a positive phrase or flag correlation means it pays for the tic or the fake.

judge_trust

Can the judge be trusted? The reward is only as good as the judge.

format_judge_trust

format_judge_trust(report: dict[str, Any]) -> str

judge_probes

judge_probes(
    rows: Sequence[dict],
    judge: Callable[[dict], Any],
    probes: str | Sequence[str] = 'all',
    rubric: str | None = None,
    sample: int = 40,
    seed: int = 0,
    concurrency: int = 8,
    flip_flag: float = 0.1,
) -> dict[str, Any]

Try the reward hacks a policy finds first on the judge, on purpose. Each probe in probes ("all" or names from PROBES) mutates up to sample graded rows one way and re-judges them. An additive probe (filler, the rubric’s words, a success claim, the ask echoed, a sycophantic opener) reports exploit_rate: the share of originally failing replies that pass once the text is added. A replacement probe (a well-formed tool call with empty arguments, a refusal) reports the share of replies that pass with the content gone. rubric is the text the keyword probe draws words from; without it the row’s system prompt is used. A probe that applies to no row is skipped with the reason. Returns per-probe counts and rates, exploitable_by (probes at or over flip_flag, FLIP_FLAG by default), and one warning per exploit.

judge_trust

judge_trust(
    rows: Sequence[dict],
    judge: Callable[[dict], Any] | None = None,
    gold: str = 'gold_reward',
    sample: int = 40,
    seed: int = 0,
    concurrency: int = 8,
    probes: str | Sequence[str] | None = None,
    rubric: str | None = None,
    min_agreement: float = 0.8,
    min_kappa: float = 0.6,
    allow_model_gold: bool = False,
    length_gap_flag: float = 0.15,
    flip_flag: float = 0.1,
) -> dict[str, Any]

The judge-trust report. See the module docstring. The floors and flags are keywords with their defaults in whileai.simulations.defaults: min_agreement (0.8, the human-human agreement of MT-Bench, arXiv:2306.05685), min_kappa (0.6, Landis and Koch “substantial”), length_gap_flag (0.15) and flip_flag (0.10). rows carry the judge’s reward; rows that also carry gold (0/1, default gold_reward) feed the agreement, held-out, and length checks. Pass judge to add the perturbation checks, which call it on up to sample rows twice more. probes="all" (or a list of names from PROBES) adds judge_probes, one more pass over the sample per probe; rubric feeds the keyword probe. ok is true only when a gold-labeled check ran against a person’s labels, the Wilson lower bound of agreement reached min_agreement, kappa reached min_kappa, and nothing else was flagged. With no labels every check has n=0, so ok is false with a warning saying the judge is unmeasured, not failed. gold_kind in the report says where the labels came from; model or unknown gold makes ok false with the reason unless allow_model_gold=True. The keys a caller reads first: ok (measured and clean), agreement["agreement"] and agreement["ci95"] (the number and its Wilson interval, not ci), agreement["n"] (labels compared), gold_kind, and warnings, where every line names its own fix. format_judge_trust(report) prints the whole thing.

judging

One judge contract for grading and evaluation, and the loop around it.

ScoredData

ScoredData(
    rows: list[dict],
    run_id: str,
    source: str,
    judge_name: str,
    model: str | None = None,
)

Scored trajectories: the one representation grade and eval share. Iterates as plain dicts, so it feeds simulate(traces=...), mine_traces, export_training and JSONL writers directly — no conversion scripts. .rows is a RowList: a list that also answers to being called, so both scored.rows and scored.rows() give the scored rows. SimulationData.rows, what simulate() returns, behaves the same way, so the two spellings are interchangeable across simulate() -> run_judge(). .warnings is the list of hollow-run notes run_judge filled; print it before reading any number.

ScoredData.agreement

agreement(
    self,
    gold: str | Sequence[dict] = 'gold_reward',
    reward: str = 'reward',
) -> dict[str, Any]

Agreement of this run’s rewards with a trusted label. See judge_agreement.

ScoredData.failed_traces

failed_traces(self) -> list[dict]

Failures, ready to hand to simulate(traces=...).

ScoredData.partials

partials(self) -> list[dict]

Rows with a continuous reward strictly between 0 and 1. The scalar lane: 1 pass, 0 fail, partials here, None unjudged - every contract-legal reward is visible in exactly one view.

ScoredData.push

push(self, name: str, **kwargs: Any) -> dict

Upload the scored rows to the platform: push_rows(self.rows, name, ...). Same keywords as push_rows (gate=, mode=, agent=, purpose=, parent=, endorsed=, strict_hacks=). The graded copy is what a gated RL push needs, and SimulationData.push cannot see it: grade(judge=) leaves the run’s trajectories ungraded on purpose.

ScoredData.select_by_reward_range

select_by_reward_range(
    self,
    lo: float,
    hi: float,
    inclusive: bool = True,
) -> list[dict]

Rows whose numeric reward falls in [lo, hi] (or (lo, hi)).

ScoredData.select_for_preference

select_for_preference(
    self,
    max_pairs_per_prompt: int = 1,
    min_margin: float = 1.0,
    length_match: bool = True,
) -> tuple[list[dict], dict[str, Any]]

Chosen/rejected pairs from same-task contrast. Failures earn here.

ScoredData.select_for_rl

select_for_rl(
    self,
    target: int = 1000,
    lo: float = 0.2,
    hi: float = 0.8,
    has_tools: bool = True,
) -> tuple[list[dict], dict[str, Any]]

Whole mixed-reward groups for RL; groups never split. lo and hi default to DIFFICULTY_BAND (0.2, 0.8), the same band select_for_rl and optimize use; they used to be 0.3 and 0.7 here alone.

ScoredData.select_for_sft

select_for_sft(self, target: int = 1000) -> tuple[list[dict], dict[str, Any]]

Diverse correct demonstrations: 1-labeled, deduped by behavior.

ScoredData.unjudged

unjudged(self) -> list[dict]

Rows the judge could not score. Never treated as failures.

build_preference_pairs

build_preference_pairs(
    rows: Sequence[dict],
    max_pairs_per_prompt: int = 1,
    min_margin: float = 1.0,
    length_match: bool = True,
) -> tuple[list[dict], dict[str, Any]]

Same-task chosen/rejected pairs for preference training (DPO-style). A pair exists only where the same prompt has two trajectories whose rewards differ by at least min_margin — the contrast is the training signal, so failures are supply here, not waste. The default 1.0 pairs 1-labeled with 0-labeled rows only; 0.5 also admits partial-credit rows against a full pass or fail. Rows without a valid judge result never pair. Each pair keeps what the trainer and the reviewer need to trust it:

chosen_score / rejected_score / margin: the raw scores and their gap, so a margin-aware loss (Llama 2 style) can use them and a reviewer can see how far apart the two really are.
chosen_model / rejected_model / same_policy: which policy produced each side. Preference data works best when both sides come from the policy being trained (Tulu 3, rlhf-book ch. 11); a mixed pair is still a pair, but it is labeled as off-policy.
length_delta: chosen reply chars minus rejected. DPO exploits a length gap faster than it learns the behavior (rlhf-book ch. 8), so with length_match=True each chosen row takes the rejected row closest to it in length, and the report says how often chosen is still the longer side.

Returns (pairs, report); each pair carries both parents’ lineage.

evaluate

evaluate(
    rows: Sequence[dict] | Any = None,
    judge: Callable[[dict], Any] | None = None,
    grader: Callable[[dict], Any] | None = None,
    model: str | None = None,
    eval_set: Sequence[Any] | None = None,
    judge_name: str | None = None,
    run_id: str | None = None,
    concurrency: int = 8,
    timeout: float | None = None,
    scale: tuple[float, float] | None = None,
    tools: Sequence[dict] | Sequence[str] | None = None,
) -> ScoredData

Judge held-out rollouts under the exact contract grade uses. Read the result’s warnings before its numbers: no rollout called a tool, a declared tool none touched (tools=, or pass the SimulationData as rows), a marker that fired on no row. Same engine, same schema; only the lineage source differs. Feeding evaluate(...).traces to simulate(traces=...) is the loop-closing move. grader= is the doctrine-sketch name for the judge callable; either spelling works, not both. eval_set= (prompt strings or rows) checks that the rollouts actually cover the frozen evaluation set and reports the gap on the result’s eval_coverage instead of letting a silent partial eval pass as a full one.

normalize_judge_result

normalize_judge_result(
    raw: Any,
    scale: tuple[float, float] | None = None,
) -> dict[str, Any]

Coerce one judge return into the contract; never invent a reward. scale=(lo, hi) reads the judge’s number as a rating on that scale (a 1 to 5 Likert, a 0 to 10 score): the row’s reward is the rating mapped onto [0, 1] and judge_meta keeps rating and scale. A rating outside the scale is a contract break, as a reward outside [0, 1] is without one. A dict may carry the number as rating instead of score when a scale is set.

run_judge

run_judge(
    rows: Sequence[dict] | Any,
    judge: Callable[[dict], Any],
    judge_name: str | None = None,
    source: str = 'grade',
    model: str | None = None,
    run_id: str | None = None,
    concurrency: int = 8,
    timeout: float | None = None,
    version: str | None = None,
    scale: tuple[float, float] | None = None,
    tools: Sequence[dict] | Sequence[str] | None = None,
) -> ScoredData

Score trajectories with any judge. Originals are left unmodified. tools= is the agent’s declared tool list (or names); with it the result’s warnings also say which declared tools no rollout called. Passing the SimulationData itself as rows supplies it. Each scored row is a copy of the input row plus reward, reason, judge_status, judge_meta, and a lineage record naming the scoring run, its source (grade or eval), the judged model, and the parent trajectory. Rows whose judge result breaks the contract keep reward=None and a non-ok status; they are counted, not hidden. version names the judge’s version (model, prompt hash, whatever would change its labels); it lands in lineage.judge_version and reads back as Judgment.scorer.version.

labels

Human labels on rows: who said what, and do they agree.

annotator_agreement

annotator_agreement(rows: Sequence[dict]) -> dict[str, Any]

How the annotators on gold_labels agree with each other. per_annotator: labels given and pass share. multi_labeled: rows with two or more annotators; unanimous the share of those where every label matched; kappa Cohen’s kappa when exactly two annotators labeled the same rows (pair names them), else None. disagreements lists the split rows (prompt, labels) so a person can read the ones the guideline did not settle (rlhf-book ch. 11).

attach_labels

attach_labels(
    rows: Sequence[dict],
    labels: Any,
    annotator: str | None = None,
    kind: str = 'human',
    replace: bool = False,
) -> tuple[list[dict], dict[str, Any]]

Write hand labels onto rows (in place) and return (rows, report). labels is a JSONL path, a list of dicts, or a {key: label} mapping. A dict label names its row by key / rollout_id, scenario_id + rollout_index, or prompt (+ final_text), and carries label (or reward / gold_reward: 0 or 1), and optionally annotator, note, ts. annotator here is the default for labels that name none. kind is recorded on each label ("human"; "model" for a stronger model’s labels). Each row gains gold_labels (every label, appended unless replace), gold_reward, the majority of its labels, and gold_kind: "human" when every label on the row is a person’s, else the other kind. judge_trust and judge_agreement only count human gold as a measurement of the judge. A tie leaves both unset. Labels that name no row, or carry no 0/1 value, are counted and listed.

logprobs

What the policy’s own log-probabilities buy you.

logprob_report

logprob_report(rows: Sequence[dict]) -> dict[str, Any]

Coverage and shape of the captured logprobs. mean_token_logprob is total logprob over total tokens. The per-row quantiles are of each row’s own mean, so one long rollout does not dominate. corr_reward_confidence is Pearson between the 0/1 reward and the per-row mean over graded rows: a strong positive value says the judge rewards fluency, not behavior.

mean_kl

mean_kl(
    rows: Sequence[dict],
    ref: str | Sequence[dict] = 'ref_logprob',
) -> dict[str, Any]

Sampled KL(pi || pi_ref) per generated token, overall and per task. ref is either a key on the same rows holding the reference model’s summed logprob over the same tokens (default ref_logprob), or a second row list scored under the reference, matched by rollout id, scenario id plus rollout index, or prompt plus final text, carrying logprob. Rows missing either side are skipped and counted. Per task the estimate pools tokens across that task’s rollouts, which is what a per-task difficulty record wants.

staleness_report

staleness_report(
    rows: Sequence[dict],
    base_model: str | None = None,
) -> dict[str, Any]

Which policies produced these rows, and can an update still use them. rlhf-book ch. 6 (asynchronous RL, truncated importance sampling): rows sampled by an older policy are usable when the row carries the sampler’s version and its logprobs so the ratio can be formed; rows from an unknown sampler are not. versions counts rows per policy_version (model_version when the row predates it); base_model names the model about to be trained, and rows whose model_version differs are stale. Coverage says how many rows carry sampling, logprob and token_logprobs.

markers

Stock behavioral markers for the over-optimization signatures (book ch. 14).

behavioral_markers

behavioral_markers(
    rows: Sequence[dict],
    names: Sequence[str] | None = None,
) -> dict[str, float]

Rate of each stock marker over rows (fraction of rollouts that trip it). The over-optimization dashboard in one call. Deprecated: use score.style.style_report for the delta-ready view.

format_markers

format_markers(report: dict[str, float]) -> str

One line per marker, highest rate first.

mark_rows

mark_rows(
    rows: Sequence[dict],
    names: Sequence[str] | None = None,
    extra: dict[str, Callable[[dict], float]] | None = None,
) -> list[dict]

Return copies of rows with the stock markers merged into each row’s markers dict, ready for marker_summary / delta_report. extra adds custom named detectors row -> value. Existing marker values are kept; stock names overwrite only themselves. Deprecated: presence polarity (1 = tic present) reads a delta_report paired comparison backwards. Use score.style.style_markers.

row_markers

row_markers(row: dict, names: Sequence[str] | None = None) -> dict[str, int]

The stock markers for one row’s final text.

optimize

Concentrate a big simulated batch into the dataset post-training needs.

filter_rl_rows

filter_rl_rows(
    rows: Sequence[dict],
    has_tools: bool = True,
) -> tuple[list[dict], dict[str, Any]]

Split keep/drop. Does not mutate rows.

group_signal

group_signal(
    rows: Sequence[dict],
    lo: float = 0.2,
    hi: float = 0.8,
) -> dict[str, Any]

Within-ask contrast. Signal is a group whose k rollouts disagree. A grouped RL update learns from a mix of 0 and 1 on the same ask, ideally with pass rate p in [lo, hi]. Unanimous groups are dead gradient. Groups of one rollout cannot mix and are counted separately, not blamed.

next_round

next_round(
    prior: Sequence[dict],
    tasks: Sequence[Any] | None = None,
    lo: float = 0.2,
    hi: float = 0.8,
) -> dict[str, Any]

The prompt set for the next round, from the last round’s graded rollouts. A round trained on the file it started from keeps paying for groups that give no gradient: at a 0.65 training reward about half the groups are all-pass or all-fail. The band is the fix the book already names (rlhf-book ch. 7: filter to the 20-80% band; ch. 6, DAPO’s dynamic sampling drops groups with no contrast), applied to what the current policy does rather than what the base did. prior is round N’s graded rollouts (simulate(tasks=..., repeats=k) on the round-N policy, or the trainer’s own sampled rows); each task’s pass rate over them decides: inside [lo, hi] it is kept, above hi it is solved and dropped, below lo it is unsolved and dropped. tasks restricts the candidates (rows, task dicts with a prompt, or prompt strings); a task with no prior rollouts is unknown and kept, since nothing says it is flat. Returns tasks (one representative row per kept task: the prior row, with calibration.pass_rate and the band), the counts kept, dropped_solved, dropped_unsolved, unknown, pass_rates per task, band, from_policy (the policy versions the prior rows came from) and prompt_set_sha: the identity of the kept set, for lineage on the run. Push the kept rows as the next train set with parent= the last one.

optimize

optimize(
    source,
    mode: str | None = None,
    target: int = 1000,
    output: str | None = None,
    band: tuple[float, float] = (0.2, 0.8),
    enforce_band: bool = True,
    select: str = 'top_per_prompt',
    min_reward: float = 1.0,
    endorsed: Sequence[str] = (),
    truncated: str = 'drop',
    order: str = 'spread',
    audit: dict[str, Any] | None = None,
) -> tuple[list[dict], dict[str, Any]]

One call after grading: concentrate for the post-training target. source is a SimulationData, a row list, or a JSONL path. mode defaults to the data’s own mode: "sft" picks diverse correct demonstrations (select and min_reward as in select_for_sft), anything else keeps whole mixed RL groups inside the difficulty band (default 20%-80% pass rate; enforce_band=False only ranks out-of-band asks last; order is "spread" across pass rates or "middle" first, see select_for_rl). endorsed names what the reward should track (feature-name substrings such as "tool:lookup_order"), so the RL report’s hack_scan can call a shortcut a hack. Returns (rows, report); writes output when given, or <name>.<mode>.jsonl next to a path source. Never overwrites the source file unless output names it explicitly.

recommend(
    tools: Sequence[dict] | None = None,
    policy: str = '',
    system_prompt: str | None = None,
    mode: str = 'sft',
    target: int | None = None,
    mixed_rate: float = 0.5,
) -> dict[str, Any]

How much data this agent needs, from its own grid. No guessing. system_prompt= is the same text under simulate’s spelling; policy= and system_prompt= are interchangeable here as there. Grounded two ways: the agent’s measured covering grid (every cell wants SATURATION_COPIES visits, and selection wants surplus to choose from), and published post-training practice (curated agent SFT lands at 500 to 2,000 trajectories: FireAct 500, LIMA 1,000, AgentTuning 1,866; agent RL uses 8 to 16 rollouts per prompt and drops all-pass/all-fail groups: DAPO 2025, Skywork-OR1 2025). Returns the numbers plus simulate_kwargs ready to splat, and reasoning lines that show the arithmetic.

select_for_rl

select_for_rl(
    rows: Sequence[dict],
    target: int = 1000,
    lo: float = 0.2,
    hi: float = 0.8,
    enforce_band: bool = True,
    has_tools: bool = True,
    dedupe: bool = True,
    drop_truncated: bool = True,
    endorsed: Sequence[str] = (),
    truncated: str = 'drop',
    order: str = 'spread',
    prior: Sequence[dict] | None = None,
    audit: dict[str, Any] | None = None,
) -> tuple[list[dict], dict[str, Any]]

Whole mixed groups up to roughly target rows. Groups never split. audit is an audit_grades report on these rows’ verifier; when it found the verifier rejecting right answers more than FN_WARN of the time, hygiene_warnings says to fix the verifier before training on the selection (#255). prior is the previous round’s graded rollouts: tasks the round-N policy already solves (pass rate above hi on prior) or never solves (below lo) are dropped before anything else, so round N+1 trains on what that policy gets right 20-80% of the time rather than on the file round 1 started from (next_round; rlhf-book ch. 7). The report’s prior block counts kept, dropped_solved, dropped_unsolved and unknown. truncated says what happens to a rollout cut at the token cap (rlhf-book ch. 6, DAPO’s overlong handling; ch. 7 overlong filtering): "drop" removes it (the default; drop_truncated=False is the old spelling of "keep"), "keep" leaves it in with overlong=True and its own reward, riding with its ask rather than deciding it (the ask is unanimous, in band and ranked exactly as under "drop", so "keep" never returns fewer rows than "drop"; a cut rollout’s reward is not the contrast an ask is kept for), and "penalize" keeps it as a failure that does count: reward 0, the judged score under reward_before_penalty, so running past the cap is a negative signal instead of a rollout that vanished. A conduct-grade advisory 0.5 for truncation is unusable under "keep" and a 0 under "penalize". After the row gates, duplicate and truncated rollouts (dedupe, truncated), the unanimous trim, and (enforce_band) the difficulty band, remaining asks are taken round-robin across observed fault kinds, so the dataset keeps a grounded spread of no-fault, miss, timeout, and already-done situations rather than one over-represented failure. Within a fault kind, order="spread" (the default) takes asks round-robin across their pass rates, so a 25% ask, a 50% ask and a 75% ask are picked in turn with no preference for the middle (rlhf-book ch. 7 filters to the 20-80% band and stops there; nothing in it says 50% is better than 30%). order="middle" is the older ranking by closeness to a 50% pass rate. The last group may overshoot target; an RL update wants the complete group or none of it. enforce_band=False keeps out-of-band asks and only ranks them last. The report’s hack_scan block is the reward-hack scan over the selection (hack_scan: what separates reward within an ask, against a permutation floor; endorsed names what it should be), correlations the older pooled scan. Reward tracking a shortcut is a judge problem, flagged in hygiene_warnings, not pruned. Selected rows are stamped in place with the calibration measured on the rows as they arrived, before dedupe and the trims: the pass rate over the k repeats the grader saw is the task’s difficulty, and re-measuring it on the survivors would report the post-dedup k under that name. publish_gate keeps the carried stamp. The k-way reliability numbers do not survive the prune, and hygiene_warnings says so when they were available before it.

select_for_sft

select_for_sft(
    rows: Sequence[dict],
    target: int = 1000,
    select: str = 'top_per_prompt',
    k: int | None = None,
    min_reward: float = 1.0,
    seed: int = 0,
) -> tuple[list[dict], dict[str, Any]]

Diverse correct demonstrations, at most target rows. Imitation clones what it sees, so only rows whose reward reaches min_reward (default 1.0: judge-approved) and that are not junk qualify; unanimity is not a problem here. A grader with partial credit ranks by its score: lower min_reward to admit it. select is the rejection-sampling rule (rlhf-book ch. 9, “Scoring Completions”): "top_per_prompt" keeps each prompt’s highest-reward completion and then round-robins across behavior signatures (tool sequence, argument provenance, outcome shape) so every distinct way of being right appears before any repeats; "top_k_overall" keeps the k highest-reward completions across all prompts, several per prompt allowed; the two random_* rules are the book’s control (same counts, seeded random picks) so a claimed gain from selection can be checked against chance. k defaults to target.

trim_out_of_band

trim_out_of_band(
    rows: Sequence[dict],
    lo: float = 0.2,
    hi: float = 0.8,
    min_k: int = 2,
) -> tuple[list[dict], dict[str, Any]]

Difficulty band filter. Nothing to do with topic or relevance. “Out of band” here means outside the difficulty band [lo, hi] (default DEFAULT_BAND, 0.2 to 0.8): an ask is dropped when its pass rate over k >= min_k rollouts is too high (the policy almost always solves it) or too low (it almost never does), because either way it carries little gradient per rollout. It does not read the prompt, the topic, or the tools; a perfectly on-topic ask is dropped for being too easy, and an off-topic one the policy passes half the time is kept. Junk rows are a separate filter (is_incomplete_junk, applied by optimize), and nothing here filters by topic at all. Unanimous asks are trim_unanimous_groups’s job and are left alone here; singles always stay.

trim_unanimous_groups

trim_unanimous_groups(
    rows: Sequence[dict],
    min_k: int = 2,
) -> tuple[list[dict], dict[str, Any]]

Drop asks whose k >= min_k rollouts all landed 0 or all landed 1. The basic optimizer from the working decision: trim zeros and ones from tasks, then rerun the simulator and check the variance. Groups smaller than min_k (unique-situation runs) always stay; trimming them would gut an explore dataset, and they carry no group gradient either way.

pairwise

Pairwise judging: which of two replies to the same request is better, asked both ways round.

judge_pairs

judge_pairs(
    pairs: Sequence[dict],
    judge: Callable[[dict, dict], Any] | None = None,
    spec: str | None = None,
    swap: bool = True,
    concurrency: int = 8,
    api_key: str | None = None,
    examples: int = 10,
    position_flip_flag: float = 0.2,
    prefers_rejected_flag: float = 0.2,
) -> tuple[list[dict], dict[str, Any]]

Ask a judge which side of each pair is better, both ways round. judge(a_row, b_row) returns \{"winner": "A" | "B" | "tie", "reason"\} (or that JSON as a string); without one the hosted model judge from pairwise_judge(spec) is used. With swap=True each pair is judged twice with A and B exchanged; a pair the judge decides differently in the two orders is recorded as a tie with position_consistent=False. Writes on each pair (in place, and returned): pairwise with winner ("chosen" | "rejected" | "tie" | None when the judge failed), position_consistent, reasons, judge; and tie (bool). Report: position_flip_rate (position bias: the judge’s answer changed with the order), tie_rate, agrees_with_scores (the pairwise winner is the pointwise chosen), prefers_rejected (the two disagree outright, the rows a person should read), failed. A position_flip_rate at or over position_flip_flag (POSITION_FLIP_FLAG, 0.2: Zheng et al. arXiv:2306.05685 measured 35% of GPT-4 verdicts flipping with the order) and a prefers-rejected share at or over prefers_rejected_flag each add a warning.

pairwise_judge

pairwise_judge(
    spec: str | None = None,
    api_key: str | None = None,
    prompt: str | None = None,
    policy: str = '',
    tools: Sequence | None = None,
    timeout: float = 120,
    max_tokens: int = 120,
    request_chars: int = 4000,
) -> Callable[[dict, dict], Verdict]

A model judge for judge_pairs: judge(a_row, b_row) -> \{"winner": "A" | "B" | "tie" | None, "reason": str\}. spec is a backend spec (default the hosted judge); prompt replaces the pairwise system prompt. The judge’s name is <model>@<prompt sha> so a prompt edit is a new judge. max_tokens is the judge’s reply budget and request_chars how much of the request it is shown.

passat

pass@1, pass^k and pass@k from the same graded groups.

PassAt

PassAt(
    k: int,
    pass_at_1: float | None,
    pass_pow_k: float | None,
    pass_at_k: float | None,
    n_groups: int,
    n_rows: int,
    n_groups_at_k: int = 0,
    n_groups_imputed: int = 0,
    per_task: dict[str, float] = <factory>,
    note: str = '',
    config: dict[str, Any] = <factory>,
    ci95: tuple[float, float] | None = None,
    pass_pow_k_ci95: tuple[float, float] | None = None,
    pass_at_k_ci95: tuple[float, float] | None = None,
) -> None

pass@1 / pass^k / pass@k over graded groups. See module docstring. Every field, and the name it prints as in str(...). The printed line and the attribute are not spelled the same: pass^k is pass_pow_k (not pass_hat_k), and the print says both once so the attribute is readable off it.

attribute	prints as	what it is
`k`	`k=4`	draw size the k-way numbers used
`pass_at_1`	`pass@1`	mean per-task pass rate, the headline
`pass_pow_k`	`pass^4`	chance all k repeats pass (reliability)
`pass_at_k`	`pass@4`	chance at least one of k passes
`headroom`	`headroom`	property: pass@k minus pass@1
`ci95`	`[lo..hi]`	task-bootstrap interval on pass@1
`pass_pow_k_ci95`	`[lo..hi]`	same for pass^k; `None` under 3 groups
`pass_at_k_ci95`	`[lo..hi]`	same for pass@k; `None` under 3 groups
`n_groups`	`N groups`	tasks pass@1 averaged over
`n_rows`	not printed	graded rows behind those tasks
`n_groups_at_k`	not printed	tasks the k-way numbers used
`n_groups_imputed`	not printed	short unanimous tasks counted in
`per_task`	not printed	`{task key: pass rate}`, a dict
`note`	tail of the line	why a number is missing, and the fix
`config`	token-cap share	how the rows were made (`run_config`)

to_dict() uses these same keys, with headroom added and the intervals as lists.

pass_at

pass_at(
    rows: Sequence[dict] | Any,
    k: int | None = None,
    min_k: int = 4,
    unanimous_short: bool = False,
) -> PassAt

pass@1, pass^k and pass@k from graded rows, grouped by task. A task is a situation, not a string. Rows group under task_key: the engine’s scenario_id when the row has one, else task_id, else the prompt text. In mode="rl" the repeats of one opener share a scenario_id, and so do the textured phrasings of one situation, so those phrasings pool into one task on purpose: the question is whether the agent handles the situation, not one wording of it. compare_runs, delta_report, eval_variance, curriculum and group_signal count tasks with the same key, so pass_at(rows).n_groups and delta_report(...)["n_paired_tasks"] agree on the same rows. Only binary reward (or qwen_reward) rows count; partial and unjudged rows are skipped, the same rule group_signal uses. k defaults to the smallest group of two or more repeats, so every such group contributes to the k-way estimators; groups with fewer than k graded repeats are left out of pass^k and pass@k (counted in n_groups_at_k). pass@1 always averages every group. Below min_k repeats (ROLLOUTS_PER_TASK, 4: the smallest k tau-bench and tau2-bench report a pass^k on, arXiv:2406.12045 and arXiv:2506.07982) the k-way numbers are None with a note instead of a number too noisy to act on. Pass k= to choose the draw size yourself. unanimous_short=True counts a unanimous group shorter than k as if it stayed unanimous (pass^k and pass@k equal to its pass rate, 1 or 0). That is the assumption a successive-allocation run stopped on, and leaving those groups out would score only the tasks that split and inflate the headroom. Mixed short groups still stay out. .config says how the rows were produced (run_config): task count, k, temperature, max_tokens, policy and judge versions, prompt hash, with a mixed list naming any the rows disagree on.

preflight

Inspect the agent before spending simulation budget, and report after.

classify_failure

classify_failure(row: dict) -> str | None

Fixed-vocabulary class for a failing row, from its reason and shape. Returns None for passing or unlabeled rows and for failures the heuristics cannot place (leave those for a person, do not guess).

coverage_gap

coverage_gap(
    asks: Any,
    tools: Sequence[dict],
    system_prompt: str = '',
    rows: Sequence[dict] | None = None,
) -> dict[str, Any]

Which parts of an agent’s policy the asks you already send never reach. asks is what a suite asks the agent: a list of prompt strings, a list of rows carrying prompt, or a path to a .py or .jsonl file holding either. The axes come from build_dimensions, the same grid simulate covers, so the answer is in the engine’s own vocabulary: which tool, which policy rule, what stance the person takes, what the world looks like, what condition the tool is in, what happened before. Each ask is placed on the axes it touches with text heuristics, not a model: the tools its words name or imply, the rule clauses it shares words with, and the stance its words show. untested_rules and untested_tools are the parts of the policy no ask reaches, which is the list worth reading. Two axes (world_state, tool_condition) cannot be read from an ask at all: a prompt never says the order is missing or the tool timed out, so a hand-written suite leaves them at one point and notes says so. With rows (graded rollouts from a run) the report also checks the world side: rules whose rows all ended in the same tool fault are rules the asks reach but the fixtures never let happen.

dataset_report

dataset_report(
    rows: Sequence[dict],
    tools: Sequence[dict] | None = None,
    system_prompt: str = '',
    hard_share_floor: float = 0.3,
) -> dict[str, Any]

One report a developer reads after simulate/grade: size, signal, mix. hard_share_floor (HARD_SHARE_FLOOR, 0.3) is the share of hard- tier rows under which the set is called easy.

format_coverage_gap

format_coverage_gap(report: dict[str, Any]) -> str

The gap report as the block a person actually reads.

preflight

preflight(tools: Sequence[dict], system_prompt: str = '') -> dict[str, Any]

Spec-quality report for an agent. Report only; nothing is changed. warnings is the list a developer should read before generating thousands of rows; cells is the covering-grid size the same way recommend counts it.

privileged

Did the agent say what only the teacher was told?

format_leak_report

format_leak_report(report: dict[str, Any]) -> str

One line per fact, the summary first.

leak_report

leak_report(rows: Any, min_len: int = 12) -> dict[str, Any]

Which rows quote their own privileged block in the agent’s text. Takes the SimulationData itself, data.trajectories, or any list of rows. Given the data object it reads the trajectories, which still carry the block; data.rows() is the scrubbed export and checks nothing (the report says so). Checks every row that carries privileged (reference, principle, and every string in hidden_state at least min_len characters long) against the final reply and every assistant turn. Returns n_rows, n_checked, n_leaked, rate (over checked rows), checked (False when no row carried the block, so the result is vacuous), leaked (up to 20 rows: scenario_id, rollout_index, field, needle) and summary. Does not mutate rows. Pass data.trajectories, not data.rows(): the export scrubs privileged at any depth, so exported rows carry nothing to check and the report is vacuous. When it can tell the rows came through the export, summary says so and names the accessor to use instead.

publish_gate

The gate a dataset passes before it leaves for the platform.

PublishGateError

The dataset must not be published as it stands. The message says why.

calibrate

calibrate(
    rows: Sequence[dict],
    policy: PolicyRef | dict | str | None = None,
    model: str | None = None,
    ref: str | Sequence[dict] | None = None,
) -> dict[str, Any]

Stamp calibration on every graded row, in place. The per-task pass rate is over the binary rewards grouped by prompt, the same grouping group_signal and pass_at use. Rows without a 0/1 reward are left alone and counted. Returns a report with the number of tasks and rows stamped plus the pass_at summary. ref (a key holding the reference model’s summed logprob, or rows scored under it) fills mean_kl per task from the captured logprobs; see mean_kl. A row whose carried stamp counts more repeats than these rows hold keeps it (n_carried in the report): optimize(mode="rl") drops duplicate trajectories, and recomputing here would report the post-dedup k as the policy’s pass rate over k repeats. The producing policy and mean_kl are still filled in from this call.

publish_gate

publish_gate(
    rows: Sequence[dict],
    mode: str | None = None,
    band: tuple[float, float] = (0.2, 0.8),
    policy: PolicyRef | dict | str | None = None,
    model: str | None = None,
    strict: bool = True,
    endorsed: Sequence[str] = (),
    strict_hacks: bool = False,
) -> dict[str, Any]

Check, calibrate, and report. Raises PublishGateError when strict and the rows are RL-shaped but ungraded or carry no mixed group, or when strict_hacks and hack_scan (with endorsed naming what the reward should track) finds the reward best explained by something else. Never mutates anything except the calibration stamp. judge_trust in the report is the summary grade stamped on the rows when they carried human labels, else None.

quality

Second-pass conversation quality ranker. Scores rows; does not rewrite them.

rank_rows

rank_rows(rows: Sequence[dict]) -> list[dict]

Score each row in place. Returns the same list when given a list.

score_row

score_row(row: dict) -> dict[str, Any]

Score one row. Returns quality, quality_reason, quality_scores. No mutate.

reference

Score rollouts under a reference model, so mean_kl has its other side.

reference_logprobs

reference_logprobs(
    source,
    ref: str,
    system_prompt: str | None = None,
    tools: Sequence[dict] | None = None,
    api_key: str | None = None,
    concurrency: int = 4,
    timeout: float = 600.0,
    chat_template_kwargs: Mapping[str, Any] | None = None,
    transport: Callable[[dict], dict] | None = None,
) -> dict[str, Any]

Stamp ref_logprob on every row: the reference’s summed logprob over the tokens the policy generated. Rows are modified in place; the report says what was scored. ref is a backend spec, vllm:<model>@<base_url>; on the platform’s serving endpoint <model> is the base by its own name (Qwen/Qwen3-4B: the reference of an SFT/GRPO/DPO run), a hosted model’s name, or run:<runId> for a finished run’s adapter, with WHILEAI_API_KEY as the key. source is a SimulationData (system prompt and tools come from its profile), a row list, or a JSONL path; pass system_prompt=/tools= for the last two so the reference sees the prompt the policy saw. chat_template_kwargs must match what the policy sampled with (\{"enable_thinking": False\} for Qwen3). Report: n_rows, n_skipped (no assistant turn or a failed call, with errors), n_tokens, model, and token_count_gap (mean |ref_n_tokens - n_tokens| over rows that carry n_tokens): near zero when the reference shares the policy’s tokenizer, which is when mean_kl is a KL and not a length artifact.

rubric

Rubrics: prompt-specific criteria as an object, a judge that scores them one by one, and a writer that drafts them.

Criterion

Criterion(
    title: str,
    description: str = '',
    weight: float = 1.0,
    kind: str = 'principle',
) -> None

One rubric item. weight is a positive magnitude; a pitfall subtracts it when the reply exhibits the mistake, a principle adds it when met, and a hard rule missed fails the whole reply.

Rubric

Rubric(
    criteria: tuple[Criterion, ...],
    source: str = 'hand',
    domain: str = '',
    notes: str = '',
    _cache: dict = <factory>,
) -> None

Rubric.checklist

checklist(self) -> str

The rubric as the judge reads it.

Rubric.score

score(self, results: Mapping[str, Any]) -> dict[str, Any]

Reward from one verdict per criterion (keyed by title or slug; a truthy value means the reply meets a hard rule or principle, or exhibits a pitfall). A missed hard rule is a 0. Otherwise the reward is the met principle weight minus the exhibited pitfall weight, over the total principle weight, clamped to [0, 1]; with no principle the reward is 1 minus the pitfall share. Criteria the judge did not answer count as not met (and not exhibited) and are listed.

attach_rubric

attach_rubric(
    rows: Sequence[dict],
    rubric: Rubric | Mapping[str, Any] | Sequence[Mapping[str, Any]] | Callable[[dict], Any],
    overwrite: bool = True,
) -> list[dict]

Put a rubric on each row’s privileged block (in place). rubric is a Rubric, its dict / list form, or row -> Rubric | None for a per-prompt rubric; None leaves that row alone.

rubric_judge

rubric_judge(
    rubric: Rubric | None = None,
    spec: str | None = None,
    api_key: str | None = None,
    prompt: str | None = None,
    policy: str = '',
    tools: Sequence | None = None,
    timeout: float = 120,
) -> Callable[[dict], dict[str, Any]]

A judge for run_judge / data.grade(judge=) that scores the rubric item by item. rubric applies to every row; without one the row’s own privileged.rubric is used and a row with none stays ungraded. The result carries reward (Rubric.score), reason, markers (rubric:<slug> = 1.0 met / 0.0 not, and for a pitfall 1.0 clean / 0.0 exhibited), criteria (the raw verdicts), rubric_version and the score breakdown. The judge’s name folds the rubric version in when one is fixed. The hosted judge scales to zero, so the first row through warms it once (warm_judge, a 600s budget) while the rest of the fan-out waits. Without that, run_judge’s eight concurrent calls all raced a container that was still loading its weights and every row came back invalid_result with a TimeoutError. Warm-up failure is not fatal: the rows are judged anyway and report the real error.

rubric_of

rubric_of(row: Mapping[str, Any]) -> Rubric | None

write_rubrics

write_rubrics(
    rows: Sequence[dict],
    spec: str | None = None,
    domain: str = '',
    n_criteria: tuple[int, int] = (3, 8),
    reference_key: str = 'reference',
    overwrite: bool = False,
    concurrency: int = 8,
    api_key: str | None = None,
    timeout: float = 120,
    writer: Callable[[str], Any] | None = None,
    max_hard: int | None = None,
) -> tuple[list[dict], dict[str, Any]]

Draft one rubric per distinct prompt with a model and attach it to every row of that prompt (privileged.rubric, source="model"). The writer sees the request, the row’s reference answer when there is one (privileged.reference or row[reference_key]), and the domain guidance you give it (the general rubric the book seeds from). Rows that already carry a rubric are skipped unless overwrite. writer(user_message) -> str replaces the model call for tests and for a writer of your own. Report: prompts seen, rubrics written, failures, mean criteria per rubric, the rubric versions. max_hard caps the hard rules a written rubric may carry: the heaviest max_hard stay hard and the rest become principles with their weight (demoted_hard in the report). A model writer marks most of what it wants as Essential, and every Essential item a reply misses is a 0, so an uncapped rubric fails rows a binary judge passes (measured live: 22 of 32 rows). None keeps what the writer wrote.

spec

Model spec as a versioned object (RLHF book ch. 17).

Spec

Spec(
    id: str,
    traits: tuple[Trait, ...] = (),
    source: dict = <factory>,
    version: str = '',
) -> None

A versioned model spec. version is derived from the content when left empty, so it is stable across processes and changes on any edit.

Spec.behaviors

behaviors(self) -> list[str]

Trait ids, in order. Hand to delta_report(must_not_regress=...).

Trait

Trait(
    id: str,
    name: str = '',
    principle: str = '',
    authority: str = 'should',
    examples: tuple[Any, ...] = (),
) -> None

One named expectation. authority is the strength (must / should / may, following the model-spec convention).

load_spec

load_spec(obj: Any, spec_id: str | None = None) -> Spec

Build a Spec from a constitution dict, a list of traits, or a path to a JSON file with either shape. A trait may be a full dict or a bare principle string.

spec_version

spec_version(spec: Spec | dict | Any) -> str

The content version of a Spec (or anything load_spec accepts).

stamp_spec

stamp_spec(rows: Sequence[dict], spec: Spec) -> list[dict]

Return copies of rows tagged with the spec they were produced or graded against: spec_id and spec_version. Provenance for the ch. 17 retention question — did adherence hold from one spec version, or model version, to the next.

stage

Stage lineage: which post-training stage consumed each row (book ch. 3).

format_stages

format_stages(report: dict[str, Any]) -> str

stage_of

stage_of(row: dict) -> str | None

The stamped stage, or None. An eval-sourced row with no stamp reads as eval (its lineage.source), so a held-out set is never mistaken for training data just because no one stamped it.

stage_report

stage_report(rows: Sequence[dict]) -> dict[str, Any]

Rows per stage, tasks per stage, and the cross-stage leaks: any task used both in eval and in a training stage (sft/rm/rl/mid). That leak means the number you report was optimized against.

stamp_stage

stamp_stage(rows: Sequence[dict], stage: str) -> list[dict]

Return copies of rows with row["stage"] = stage. stage must be one of STAGES; nothing else is touched.

stats

Confidence intervals, paired run comparison, and decontamination.

compare_runs

compare_runs(
    a: Sequence[dict],
    b: Sequence[dict],
    metric: str = 'pass_at_1',
    n_boot: int = 2000,
    seed: int = 0,
    min_paired: int = 5,
    level: float = 0.95,
) -> dict[str, Any]

Is run b different from run a on metric? Tasks the two runs share are compared as paired differences (b minus a, per task); the interval is a level bootstrap over those pairs (ci95 at the default, with level reported beside it) and the p-value is a sign-flip permutation test. With fewer than min_paired shared tasks the comparison falls back to unpaired task means and says so. verdict is one of "b_better", "a_better", "no_difference_detected": the last means the interval covers zero, not that the runs are equal. Tasks on one side only are dropped from a paired comparison, and note says how many, since a verdict over a quarter of the tasks is not a verdict over the eval. paired_share is the shared fraction of every task either run saw.

decontaminate

decontaminate(
    rows: Sequence[dict],
    against: Sequence[Any] | Any,
    n: int = 8,
    fields: Sequence[str] = ('prompt',),
    overlap: float = 0.8,
    embedder: Callable[[list[str]], Sequence[Sequence[float]]] | None = None,
    similarity: float = 0.85,
) -> tuple[list[dict], dict[str, Any]]

Drop rows whose prompt overlaps an evaluation set (rlhf-book ch. 16). against is one or more evaluation sources: row lists, JSONL paths, or platform dataset ids (ds_...). Evaluation prompts, answers and references are the texts (not the eval set’s own replies). Four rules, applied in this order, and a row flagged by one is not counted again by the next, so n_contaminated is the number of rows dropped:

same_task (n_same_task): the row’s scenario_id or task_id is an evaluation row’s. A task is a situation, not a string (task_key), so a rephrasing of an eval situation is the eval situation whatever the words say. Rows with no recorded id skip this rule.
exact (n_exact): one of the row’s fields is an evaluation text verbatim after normalization (case and whitespace).
near copy (n_near): one evaluation text covers at least overlap of the row’s words with shared word n-grams (the Llama 2 rule: 8-grams, 80% of tokens). Texts shorter than n words match verbatim only.
semantic (n_semantic), only with embedder: the cosine similarity between the row’s text and an evaluation prompt is at least similarity, and the two carry different task ids or none.

The default field is the prompt, the book’s method; add "final_text" to ask the stricter question of whether replies reproduce eval answers or references. One shared n-gram is the book’s test for free-form sets. Situations written from templates share whole sentences that say nothing about which question was asked, so any-n-gram flags every row of a template-written set; the coverage rule counts a row when one eval text accounts for most of it. overlap=0 restores any-n-gram. Word overlap does not see a paraphrase. A holdout written by re-running the generator on the same briefs was 70% within 0.85 cosine of the training batch and 5 of 133 byte-identical; the 8-gram rule flagged 4 of 101 prompts and the semantic pass 16 (#286). embedder is any callable from a list of texts to one vector per text, so nothing here imports a model; with sentence-transformers:: from sentence_transformers import SentenceTransformer model = SentenceTransformer(“BAAI/bge-small-en-v1.5”) clean, report = wai.decontaminate( train, against=[holdout], embedder=lambda texts: model.encode(texts, normalize_embeddings=True).tolist(), ) A semantic flag means the two prompts read alike, not that they are the same task: “cancel one reservation” and “cancel three reservations” for different customers scored 0.932 with no shared answer. So where task identity is recorded the same_task rule decides and the semantic pass only looks across different tasks, and the report’s notes say the flag is a question to check, not a verdict. The default stays lexical: similarity 0.85 was read off BGE (unrelated prompts score about 0.55 there) and does not transfer to every model, so the pass calibrates it for yours when it can: with eval rows that carry task ids, the 99th percentile of similarity over eval-prompt pairs with different task ids is how alike distinct tasks read to this embedder, and notes says it. A threshold below that number flags tasks that merely share a domain, and the note says so when similarity is. Returns the clean rows and a report: the count under each rule, hits per field, the eval text count, and the first offenders with their coverage (or similarity for semantic hits).

detectable_effect

detectable_effect(
    n_tasks: int,
    base: float = 0.6,
    k: int = 4,
    power: float = 0.8,
    alpha: float = 0.05,
) -> float | None

The smallest gain n_tasks paired tasks can prove at power: holdout_size solved for the effect (FIXED_POINT_STEPS fixed-point steps, since the after-side variance depends on it). None below MIN_HOLDOUT_TASKS tasks.

eval_variance

eval_variance(
    *runs: Sequence[dict],
    metric: str = 'pass_at_1',
    by: str | None = None,
) -> dict[str, Any]

How much an evaluation moves when the same model is evaluated again (rlhf-book ch. 16, Evaluation). Pass each re-run’s rows as its own argument, or one row list whose rows say which run they belong to: lineage.eval_run (what simulate(runs=3) stamps), else lineage.scoring_run_id (what evaluate(run_id=) stamps), or a top-level or lineage key named by by. Each run’s metric is a mean over tasks; the report is those means, their mean, the sample standard deviation run_std, and noise_band = noise_band(run_std), 1.96 x sqrt(2) x run_std: a before/after delta with one run per side is the difference of two re-run draws, and a delta inside that band is what re-running the eval does on its own (run_std is taken as the eval’s spread; from three runs it is rough, and the note says so). run_std_by_metric reports the same floor for pass@1 and every marker shared by all runs; hand that mapping to delta_report(run_std=) so each metric uses its own re-run variance. The scalar run_std remains the selected metric’s value for callers comparing only one metric. stability places run_std on Olmo 3’s bands in points. Fewer than three runs is a difference, not a distribution; the report says so and run_std is None below two.

holdout_size

holdout_size(
    effect: float,
    base: float = 0.6,
    k: int = 4,
    power: float = 0.8,
    alpha: float = 0.05,
    before: Sequence[dict] | None = None,
    after: Sequence[dict] | None = None,
    task_std: float | None = None,
    rows: Sequence[dict] | None = None,
) -> dict[str, Any]

How many paired tasks a holdout needs to prove a gain of effect. Models the test delta_report runs: each task’s pass rate over k rollouts on each side, the delta as the mean of the paired differences, the interval from a bootstrap over tasks. The usual two-sided power calculation then gives n = ((z_\{1-alpha/2\} + z_power) * sd / effect) ** 2 with sd the standard deviation of one task’s paired difference (rlhf-book ch. 16: the point of a better eval is statistical power when comparing training runs). Where sd comes from is the whole question, and there are three ways to answer it, best first:

before and after, the graded arms of a previous eval on the same tasks (the two row lists delta_report(before, after) takes): sd is measured as the sample sd of the per-task differences, which carries the covariance that pairing buys and whatever shape the gain had. No model. sd_source is "rows" and n_paired says how many tasks it was read off.
task_std, a number you measured (the per-task sibling of delta_report’s run_std): the same quantity read off a previous delta_report: (hi - lo) * sqrt(n_paired_tasks) / 3.92 from target_ci95 and n_paired_tasks (or any metrics[...]["ci95"] with its n_paired). Agent rubrics sat near 0.38 across five lanes (#288). sd_source is "given". eval_variance’s run_std is a different number (how much a re-run moves the mean) and is not this.
Neither: the binomial model sqrt((p(1-p) + q(1-q)) / k) with p = base and q = base + effect, sd_source "model". It assumes two things it cannot check: that the gain is spread evenly across tasks, and that the two arms are independent draws (Var(A) + Var(B), no covariance term). When the gain is carried by a few tasks, most tasks are ties and the paired differences spread far wider than binomial-per-task predicts; a voice trait at 0 -> 0.127, k=4, carried by 19 of 150 tasks, measured sd 0.333 against the model’s 0.168 and needed 54 tasks where the model said 14 (#292). So the model path also returns n_tasks_concentrated, the count if the gain were carried by the fewest tasks that can carry it (each going from base to 1), and notes says which assumption is in play. On a holdout whose tasks differ in difficulty the independence assumption errs the other way: the model puts p(1-p) of variance on every task where pairing keeps each task’s own p_i(1-p_i), whose mean is p(1-p) - Var(p_i), so it asks for 1 / (1 - Var(p_i) / (p(1-p))) times the tasks pairing needs (1.19x at spread 0.2 around 0.5, 2.78x at 0.4). before alone reports the spread as base_spread and puts that ratio in notes.

before on its own (rows is the same argument under its old name) reads base and k off the data. Returns n_tasks plus the inputs, task_std, sd_source, half_width (the 95% band on the delta at that n), n_tasks_concentrated, base_spread, n_paired and notes; every key is present on every path (None or [] where it does not apply). The default answer is unchanged; the honest paths are the two that measure. The recipe that asked for this had 140 tasks at k=4 around 0.6: a band of about +-0.06, so a real 3-point gain reads no_change_detected every round. This says so before training.

marker_names

marker_names(rows: Sequence[dict]) -> list[str]

marker_summary

marker_summary(
    rows: Sequence[dict],
    names: Sequence[str] | None = None,
    n_boot: int = 2000,
    seed: int = 0,
) -> dict[str, dict[str, Any]]

metric_summary for every marker on the rows (or names). Each marker’s stats are keyed mean, ci95 (not ci), n_tasks, n_rows (not n), n_rows_at_1, n_rows_at_0, degenerate, and note or warning when there is one. ci95 is None below MIN_CI_TASKS tasks, and note then says how many tasks the marker has and how many the interval needs; a reader who sees only None cannot tell that from a bug.

metric_summary

metric_summary(
    rows: Sequence[dict],
    metric: str = 'pass_at_1',
    n_boot: int = 2000,
    seed: int = 0,
) -> dict[str, Any]

Mean over tasks with a task-bootstrap 95% interval. degenerate is set when every applicable row scored the same value: the metric has not been shown to be able to come out any other way, so ci95 is None (the way pass_at returns None below three groups) and warning says so. A marker that is silently unfireable (a key-name mismatch) and one that is genuinely always true look identical otherwise, and either one passed to must_not_regress is a guard that cannot fail (#270). n_rows_at_1 and n_rows_at_0 put the row-level split next to the mean.

task_key

task_key(row: dict) -> str

The one name every report groups a row’s rollouts under. A task is a situation, not a string: scenario_id when the row has one (the engine’s situation id, shared by the repeats of one opener and by the textured phrasings of one situation), else task_id (rows from elsewhere), else the prompt text. pass_at, compare_runs, delta_report, eval_variance, curriculum, group_signal and the exporters all count tasks with this key, so the same rows give the same task count everywhere (rlhf-book ch. 16: intervals and paired comparisons are over tasks, never rows).

style

Over-optimization signatures on replies: the things a reward pays for by accident.

refusal_report

refusal_report(
    rows: Sequence[dict],
    phrases: Sequence[str] = ("i can't help with", 'i cannot help with', "i can't assist", 'i cannot assist', "i'm unable to", 'i am unable to', "i won't be able to", "i'm not able to", 'i am not able to', "i can't do that", 'i cannot do that'),
    examples: int = 5,
) -> dict[str, Any]

Over-refusal on a benign set (rlhf-book ch. 14 “Over-Refusal”). Pass the rows whose asks the agent should have answered; the report is the share it refused anyway, with a Wilson 95% interval, the phrases that fired, and the first few refusals so a person can read them. Refusal rate on a mixed set means nothing, which is why this takes the benign rows rather than finding them.

style_markers

style_markers(
    rows: Sequence[dict],
    phrases: Mapping[str, Sequence[str]] | None = None,
) -> list[dict]

Stamp the style markers on every row’s markers (in place) and return the rows. phrases overrides or extends STYLE_MARKERS: {"no_boilerplate": [...], "no_brand_voice": [...]}. Existing markers with other names are kept.

style_report

style_report(
    rows: Sequence[dict],
    phrases: Mapping[str, Sequence[str]] | None = None,
    threshold: float = 0.3,
    n_boot: int = 2000,
    seed: int = 0,
) -> dict[str, Any]

How much of each signature the replies carry, and whether the reward pays for it. Does not mutate rows. Per marker: clean (share of rows without a hit, with a task- bootstrap 95% interval), hits (rows with a hit), top_phrases (the phrases that fired, most common first) and reward_corr (Pearson between “phrase present” and the binary reward over graded rows). A positive correlation at or above threshold is flagged: the judge is rewarding the tic, and a policy trained on these rewards will produce more of it (rlhf-book ch. 14). warnings says so in one line per flag.

trace

Did the agent fake the work? Flags read from the trajectory, not the prose.

trace_flag_report

trace_flag_report(
    rows: Sequence[dict],
    kinds: Mapping[str, str] | None = None,
    threshold: float = 0.3,
    n_boot: int = 2000,
    seed: int = 0,
    examples: int = 3,
) -> dict[str, Any]

How often each flag fires, what a reviewer should read, and whether the reward pays for it. Does not mutate rows. Per flag: n (rows it fired on), rate, examples (evidence with the ask), and reward_corr (Pearson between “flag fired” and the binary reward over graded rows); a positive correlation at or over threshold is flagged, since a judge that pays for a faked turn trains a policy to fake turns. Per marker: the clean share with a task-bootstrap interval. warnings says so, one line per flag.

trace_flags

trace_flags(
    row: Mapping[str, Any],
    kinds: Mapping[str, str] | None = None,
) -> dict[str, str]

The flags that fire on one rollout: {flag: evidence}. First match wins per flag, so the evidence points at the earliest cause.

trace_markers

trace_markers(
    rows: Sequence[dict],
    kinds: Mapping[str, str] | None = None,
    evidence: bool = True,
) -> list[dict]

Stamp the trace markers on every row’s markers (in place) and return the rows: 1.0 when the family is clean, 0.0 when a flag fired. With evidence the flags and their fragments land on the row as trace_flags for a reviewer.

​agreement

​judge_agreement

​audit

​audit_grades

​format_audit

​checklist

​expected_outcome

​outcome_check

​privileged_context

​task_checklist

​curriculum

​curriculum

​format_curriculum

​retire_solved

​delta

​delta_report

​format_delta_report

​grading

​behavior_signature

​conduct_grade

​grounding

​argument_grounding

​grounding_report

​mark_grounding

​ungrounded_arguments

​hack_scan

​format_hack_scan

​format_hack_scan_diff

​hack_scan

​hack_scan_diff

​hygiene

​dedupe_groups

​length_report

​near_duplicate_prompts

​reward_correlations

​judge_trust

​format_judge_trust

​judge_probes

​judge_trust

​judging

​ScoredData

​ScoredData.agreement

​ScoredData.failed_traces

​ScoredData.partials

​ScoredData.push

​ScoredData.select_by_reward_range

​ScoredData.select_for_preference

​ScoredData.select_for_rl

​ScoredData.select_for_sft

​ScoredData.unjudged

​build_preference_pairs

​evaluate

​normalize_judge_result

​run_judge

​labels

​annotator_agreement

​attach_labels

​logprobs

​logprob_report

​mean_kl

​staleness_report

​markers

​behavioral_markers

​format_markers

​mark_rows

​row_markers

​optimize

​filter_rl_rows

​group_signal

​next_round

​optimize

​recommend

​select_for_rl

​select_for_sft

​trim_out_of_band

​trim_unanimous_groups

​pairwise

​judge_pairs

​pairwise_judge

​passat

agreement

judge_agreement

audit

audit_grades

format_audit

checklist

expected_outcome

outcome_check

privileged_context

task_checklist

curriculum

curriculum

format_curriculum

retire_solved

delta

delta_report

format_delta_report

grading

behavior_signature

conduct_grade

grounding

argument_grounding

grounding_report

mark_grounding

ungrounded_arguments

hack_scan

format_hack_scan

format_hack_scan_diff

hack_scan

hack_scan_diff

hygiene

dedupe_groups

length_report

near_duplicate_prompts

reward_correlations

judge_trust

format_judge_trust

judge_probes

judge_trust

judging

ScoredData

ScoredData.agreement

ScoredData.failed_traces

ScoredData.partials

ScoredData.push

ScoredData.select_by_reward_range

ScoredData.select_for_preference

ScoredData.select_for_rl

ScoredData.select_for_sft

ScoredData.unjudged

build_preference_pairs

evaluate

normalize_judge_result

run_judge

labels

annotator_agreement

attach_labels

logprobs

logprob_report

mean_kl

staleness_report

markers

behavioral_markers

format_markers

mark_rows

row_markers

optimize

filter_rl_rows

group_signal

next_round

optimize

recommend

select_for_rl

select_for_sft

trim_out_of_band

trim_unanimous_groups

pairwise

judge_pairs

pairwise_judge

passat