whileai.simulations.schema

The typed row schema and its conversions. 10 public names. import whileai.simulations as wai, then wai.name.

Name	What it does
`Calibration`	Measured difficulty of one task for one student.
`Dataset`	Split membership is a dataset decision, so one task can be holdout in one dataset and training in another.
`Judgment`
`Marker`
`Rollout`	One episode.
`Task`	The situation.
`calibration_of`	The typed `Calibration` a row carries, or `None` when absent or malformed.
`from_row`	Split one flat row into its four objects.
`to_row`	The flat v1 wire row.
`validate`	Problems with one row, empty when it is fine.

Calibration

Calibration(
    task_id: str,
    student: PolicyRef,
    n: int,
    pass_rate: float,
    mean_kl: float | None = None,
    pass_rate_ci95: tuple[float, float] | None = None,
) -> None

Measured difficulty of one task for one student. Optional; only calibrate produces it. pass_rate_ci95 is the Wilson 95% interval on pass_rate from n rollouts (about +/-0.3 wide at n=8), so a band assignment can be read with its uncertainty. mean_kl is the sampled KL to a reference policy per generated token; simulate(logprobs=True) captures the student side and calibrate(rows, ref=...) fills it in.

Dataset

Dataset(
    dataset_id: str,
    spec_id: str = '',
    splits: dict[str, tuple[str, ...]] = <factory>,
) -> None

Split membership is a dataset decision, so one task can be holdout in one dataset and training in another.

Judgment

Judgment(
    rollout_id: str,
    scorer: ScorerRef,
    reward: float | None,
    status: "Literal['ok', 'missing_reward', 'invalid_result', 'error', 'timeout']" = 'ok',
    reason: str = '',
    failure_class: str | None = None,
    evidence: dict = <factory>,
) -> None

Marker

Marker(
    rollout_id: str,
    name: str,
    value: float,
    evidence: dict = <factory>,
) -> None

Rollout

Rollout(
    rollout_id: str,
    task_id: str,
    policy: PolicyRef = <factory>,
    index: int = 0,
    steps: list[Step] = <factory>,
    final_text: str = '',
    ledger: list[FaultEvent] = <factory>,
    usable: bool = True,
    unusable_reason: str | None = None,
    extra: dict[str, Any] = <factory>,
) -> None

One episode. Mutable so grading paths can attach to it in place; the no-verdict invariant is enforced at the boundary, not here.

Task

Task(
    task_id: str,
    spec_id: str = '',
    prompt: str = '',
    prefix: tuple[Message, ...] = (),
    world: World = <factory>,
    privileged: Privileged = <factory>,
    axes: dict[str, Any] = <factory>,
    behaviors: tuple[str, ...] = (),
    lineage: Lineage = <factory>,
) -> None

The situation. Identity-bearing, so nothing computed lives here: splits belong to a Dataset, difficulty to a Calibration.

calibration_of

calibration_of(row: dict) -> Calibration | None

The typed Calibration a row carries, or None when absent or malformed. publish_gate / calibrate write it as a flat dict under calibration; this is the read side. mean_kl is optional.

from_row

from_row(row: dict) -> tuple[Task, Rollout, list[Judgment], list[Marker]]

Split one flat row into its four objects. Any version, any shape.

to_row

to_row(
    task: Task,
    rollout: Rollout,
    judgments: Sequence[Judgment] = (),
    markers: Sequence[Marker] = (),
) -> dict

The flat v1 wire row. Inverse of from_row on engine rows; on other shapes it is the canonical row load_traces would produce, with the source row’s unknown keys carried along.

validate

validate(
    row: Any,
    kind: "Literal['row', 'training', 'preference']" = 'row',
) -> list[str]

Problems with one row, empty when it is fine. Never raises. Stamped rows must carry the required fields with the right types. Unstamped kind="row" rows are version 0 and only have to be non-empty dicts: nothing that works today is rejected there. Two things are reported regardless of version, because “0 validation failures” is read as a guarantee and neither case is one:

an empty dict, which carries no prompt, no messages, no verdict;
a training or preference row with no messages or no chosen/rejected. Those two kinds ask “is this a training sample”, and an unstamped dict with no conversation in it is not one whatever version it claims.

Get started

Guides

Concepts

Reference

API

whileai.simulations.schema

Calibration

Dataset

Judgment

Marker

Rollout

Task

calibration_of

from_row

to_row

validate

​Calibration

​Dataset

​Judgment

​Marker

​Rollout

​Task

​calibration_of

​from_row

​to_row

​validate

Calibration

Dataset

Judgment

Marker

Rollout

Task

calibration_of

from_row

to_row

validate