whileai.simulations.ingest

Platform calls: datasets, traces, cuts, training runs, hosted models. 39 public names. import whileai.simulations as wai, then wai.name.

Name	What it does
`agents`	Every agent on your account with counts: traces, sets by purpose, public cards.
`catalog`	The public catalog: `{"datasets": [card, ...], "agents": [...]}`.
`cut`	Cut training data out of your traces: the platform’s “Make training data” button, as one line.
`cuts`	How your traces group into prompts, and what a cut would hold.
`datasets`	List your datasets plus storage used, newest first.
`delete_dataset`	Permanently delete a dataset from your account.
`delete_empty_datasets`	Delete datasets with no stored bytes, or with `max_rows` rows or fewer when that is set (smoke runs).
`dimensions_from_traces`	Coverage axes aimed at behaviors seen in `rows`.
`drop_leaky_rows`	Kept rows plus the report.
`flaw_rows`	Rows with an observed fault or a 0 label: the next round’s traces.
`format_cuts`	What `cuts()` says, as the sentence the traces page leads with.
`format_trace_report`	The trace report as a text block, aiming stated in plain words.
`hf_publish`	Push one of your datasets to a Hugging Face dataset repo you own.
`hf_publish_run`	Push a finished training run’s LoRA adapter to a Hugging Face model repo you own, with a model card (base model, metrics, the dataset repo when the data was pushed too).
`hf_status`	Is a Hugging Face account connected to this account, and which namespaces (you plus your orgs) can it publish under?
`import_hf`	Bring one split of a Hugging Face dataset onto your account as rows, so it gets a profile (pass rate, gradient support, mixed prompts) before you train on it.
`infer_harness`	Draft a harness from observed trace rows.
`issue_delegated_credential`	Create a short-lived delegated credential for SDK or backend use.
`leakage_report`	Near-copy check of generated prompts against source traces.
`load_traces`	Normalize any supported trace source to the canonical schema above.
`mine_traces`	What the deployed agent actually did, counted for grid focusing.
`opening_share`	Share of traces whose conversation opens with the assistant.
`preview`	Three sample rows and the analyzer report for one of your datasets.
`profile`	The trainer’s numbers for one of your datasets: pass rate, gradient support, tasks with both a pass and a fail, tool use, tokens, per-task pass rates.
`publish`	Publish one of your datasets as a public card on zeroproofai.com/datasets.
`pull`	Download a dataset.
`purge_agent`	Remove an agent and everything under it: its traces, its datasets, and its registry record.
`push_file`	Upload an existing JSONL file.
`push_rows`	Upload rows as JSONL to your While account.
`refresh_delegated_credential`	Refresh a delegated credential before it expires.
`register_agent`	Create or update an agent record: the name, and optionally what it is (a line), its tool schemas, and its system prompt.
`rows_from_otel`	Conversations from an OTLP JSON batch or an iterable of spans.
`send_score`	Grade a run that has already finished — the number `cut()` filters on::
`simulate_from_traces`	Alias for `simulate(agent, traces=...)`: same grid focus and leakage gate, for callers who start from the traces.
`split_pseudo_production`	Set aside a pseudo-production slice; the rest stays for training.
`tools_from_traces`	The agent’s tool surface, read off the calls the traces contain.
`trace_report`	What these traces contain and what they will aim generation at.
`unpublish`	Take a dataset off the public catalog.
`update_dataset`	Change what a dataset is for, its mode, agent or description.

otel

OTel GenAI spans to trajectory rows: simulate(traces=...) input.

rows_from_otel

rows_from_otel(
    source: Any,
    reward_keys: Sequence[str] = ('whileai.reward',),
) -> list[dict]

Conversations from an OTLP JSON batch or an iterable of spans. One row per conversation id (trace id when no conversation id is emitted): the first user message becomes prompt, tool spans in time order become steps, later user turns become user steps, and the last assistant output becomes final_text. reward_keys names the span attributes a 0..1 reward is read from (REWARD_KEYS).

platform

While platform client: dataset upload, listing, download.

agents

agents(api_key: str | None = None) -> list[dict]

Every agent on your account with counts: traces, sets by purpose, public cards. An agent exists the moment a push names it (data.push(name, agent=...)) or a trace arrives with gen_ai.agent.name; register_agent is for attaching the spec or a description ahead of that.

catalog

catalog() -> dict

The public catalog: {"datasets": [card, ...], "agents": [...]}. No key needed.

cut

cut(
    agent: str | None = None,
    kind: str = 'rl',
    since: str = 'all',
    holdout: float | None = None,
    band: tuple[float, float] | None = None,
    name: str | None = None,
    api_key: str | None = None,
    **filters: Any,
) -> dict

Cut training data out of your traces: the platform’s “Make training data” button, as one line. Runs of the same prompt are grouped by zeroproof.scenario_id. kind="rl" keeps the prompts the agent passes some of the time and not always (20% to 80% by default, RLHF book ch. 7); kind="sft" keeps the best run of every prompt that ever passed. Either way the prompts are split into a train set and a held-out set, so a prompt you measure on is never a prompt you trained on. since is the trace window ("all", "24h", "7d"); any other keyword is a trace filter, the same ones the traces page has (model=, tool=, evalSet=, measure=). band=(lo, hi) moves the pass-rate window and holdout= the split fraction. Returns the gate’s reply with the two sets pulled out:: made = wai.cut(agent=“my-agent”, kind=“rl”) wai.pull(made[“train”][“datasetId”], “train.jsonl”) made[“holdout”][“datasetId”] # measure on this, never train on it

cuts

cuts(
    agent: str | None = None,
    since: str = 'all',
    api_key: str | None = None,
    **filters: Any,
) -> dict

How your traces group into prompts, and what a cut would hold. The numbers the platform’s traces page shows above the runs table: prompts seen, rl (the ones worth training on, with the train and holdout counts), sft, and more (prompts that need more runs). Read it before cut() when you want to know what you would get. format_cuts() prints it as the three lines the traces page shows.

datasets

datasets(api_key: str | None = None) -> dict

List your datasets plus storage used, newest first.

delete_dataset

delete_dataset(dataset_id: str, api_key: str | None = None) -> dict

Permanently delete a dataset from your account.

delete_empty_datasets

delete_empty_datasets(
    max_rows: int = 0,
    dry_run: bool = False,
    api_key: str | None = None,
) -> dict

Delete datasets with no stored bytes, or with max_rows rows or fewer when that is set (smoke runs). Permanent. Returns the ids.

format_cuts

format_cuts(report: Mapping[str, Any], agent: str | None = None) -> str

What cuts() says, as the sentence the traces page leads with. The answer, the counts behind it, and the line to run next:: 3 prompts are worth training on 9 runs · 3 train · none held out next: wai.cut(agent=“my-agent”, kind=“rl”)

hf_publish

hf_publish(
    dataset_id: str,
    namespace: str | None = None,
    repo: str | None = None,
    private: bool = False,
    wait: bool = True,
    timeout: float = 600.0,
    poll: float = 2.5,
    api_key: str | None = None,
) -> dict

Push one of your datasets to a Hugging Face dataset repo you own. The set’s purpose (train, holdout, eval) is the split; pushing the holdout set into the same repo adds a second split. Every push is a commit tagged zp-<dataset id> and the repo carries whileai.json (split -> dataset, numbers, history). Defaults: your username and a slug of the dataset name. With wait (the default) this returns the finished state {"repo", "url", "commit", "tag", "split", ...}; otherwise the pushing stamp.

hf_publish_run

hf_publish_run(
    run_id: str,
    namespace: str | None = None,
    repo: str | None = None,
    private: bool = True,
    wait: bool = True,
    timeout: float = 900.0,
    poll: float = 3.0,
    api_key: str | None = None,
) -> dict

Push a finished training run’s LoRA adapter to a Hugging Face model repo you own, with a model card (base model, metrics, the dataset repo when the data was pushed too). Private by default: it is a checkpoint, not a release. Tagged zp-<run id>.

hf_status

hf_status(api_key: str | None = None) -> dict

Is a Hugging Face account connected to this account, and which namespaces (you plus your orgs) can it publish under? Connect one on any dataset page at zeroproofai.com/platform/datasets. Returns {"connected", "username", "namespaces", "scopes"}.

import_hf

import_hf(
    repo: str,
    split: str = 'train',
    revision: str | None = None,
    config: str | None = None,
    name: str | None = None,
    purpose: str = 'train',
    mode: str | None = None,
    agent: str | None = None,
    description: str | None = None,
    max_rows: int = 100000,
    wait: bool = True,
    timeout: float = 900.0,
    poll: float = 3.0,
    api_key: str | None = None,
) -> dict

Bring one split of a Hugging Face dataset onto your account as rows, so it gets a profile (pass rate, gradient support, mixed prompts) before you train on it. Parquet, JSONL, CSV and Arrow all come in the same way. Public repos need no connected account; private ones use the Hugging Face account connected on the platform. Returns the dataset row. With wait (the default) the row is ready (or this raises with the import error); otherwise it is importing and wai.datasets() shows it settle.

issue_delegated_credential

issue_delegated_credential(
    clerk_token: str | None,
    ttl_seconds: int = 3600,
    name: str = 'sdk-default',
    timeout: float = 120,
) -> dict

Create a short-lived delegated credential for SDK or backend use. clerk_token must be a valid Clerk session token or other authenticated backend token. This helper sends that token as a bearer token to the auth endpoint to mint the delegated credential.

preview

preview(dataset_id: str, api_key: str | None = None) -> dict

Three sample rows and the analyzer report for one of your datasets.

profile

profile(
    dataset_id: str,
    force: bool = False,
    api_key: str | None = None,
) -> dict

The trainer’s numbers for one of your datasets: pass rate, gradient support, tasks with both a pass and a fail, tool use, tokens, per-task pass rates. Cached on the platform until the set changes; force=True recomputes.

publish

publish(
    dataset_id: str,
    agent: str,
    description: str | None = None,
    api_key: str | None = None,
) -> dict

Publish one of your datasets as a public card on zeroproofai.com/datasets. Cards are grouped by agent (a short name such as "airline-support"). The dataset must be finalized and hold rows. Returns the card. Anyone can then pull it with no key.

pull

pull(
    dataset_id: str,
    path: str | None = None,
    api_key: str | None = None,
) -> str | list[dict]

Download a dataset. Writes JSONL to path and returns the path, or returns the parsed rows when path is omitted. Public catalog datasets need no key; your own need the usual one. A dataset is stored as one or more parts, and the grant lists every one of them. Datasets pushed with push_rows are a single part, which is why reading only downloadUrl looked correct for so long; a dataset filled by trace ingest is one part per trace, and that path returned the first row of a 60-row dataset without saying so.

purge_agent

purge_agent(
    agent: str,
    dry_run: bool = False,
    api_key: str | None = None,
) -> dict

Remove an agent and everything under it: its traces, its datasets, and its registry record. Permanent. dry_run=True only counts. Returns {"agent", "traces", "datasets", "deleted"}.

push_file

push_file(
    path: str,
    name: str | None = None,
    api_key: str | None = None,
    parent: str | None = None,
    gate: bool = True,
    mode: str | None = None,
) -> dict

Upload an existing JSONL file. name defaults to the file name. gate=True (default) parses the file, runs publish_gate (rows get their calibration stamp; RL-shaped rows that are ungraded or have no mixed group are refused), and uploads the stamped rows. The report comes back as entry["gate"]. gate=False uploads the bytes exactly as they are on disk.

push_rows

push_rows(
    rows: list[dict],
    name: str,
    api_key: str | None = None,
    parent: str | None = None,
    gate: bool = False,
    mode: str | None = None,
    purpose: str | None = None,
    agent: str | None = None,
    description: str | None = None,
    endorsed: Sequence[str] = (),
    strict_hacks: bool = False,
    prove_effect: float = 0.05,
) -> dict

Upload rows as JSONL to your While account. Returns the registry entry, including datasetId. Pass parent (a ds_... id) when this dataset is an iteration of an existing one, so lineage shows on the platform. purpose is what the set is for on the Datasets page: "train" (the default), "holdout" or "eval"; mode is the simulation mode that made it, and is also recorded. gate=True runs publish_gate first (calibration stamp; RL-shaped rows refused when ungraded or without a mixed group) and returns its report as entry["gate"]. endorsed names what the reward should track for the gate’s hack_scan; strict_hacks=True refuses a set whose reward is best explained by something else. SimulationData.push gates by default; this row-level entry point does not, because the caller may already have run optimize. A purpose="holdout" push warns when the set is too small to prove a prove_effect gain (5 points) at 80% power.

refresh_delegated_credential

refresh_delegated_credential(
    clerk_token: str | None,
    credential: str,
    ttl_seconds: int = 3600,
    timeout: float = 120,
) -> dict

Refresh a delegated credential before it expires.

register_agent

register_agent(
    name: str,
    description: str | None = None,
    tools: list | None = None,
    system_prompt: str | None = None,
    api_key: str | None = None,
) -> dict

Create or update an agent record: the name, and optionally what it is (a line), its tool schemas, and its system prompt. Returns the record.

send_score

send_score(
    trace_id: str | None = None,
    value: float | bool | None = None,
    name: str = 'score',
    scores: Sequence[dict] | None = None,
    api_key: str | None = None,
    **fields: Any,
) -> dict

Grade a run that has already finished — the number cut() filters on:: wai.send_score(“4bf92f3577b34da6”, 1.0) # passed wai.send_score(“4bf92f3577b34da6”, 0.0) # failed wai.cut(agent=“my-agent”, kind=“rl”) # now there is something to cut A run counts as a pass at 1.0 or above, which is what puts its prompt in the 20-80% band cut(kind="rl") keeps. Send a 0-to-1 quality number as the verdict and nothing will ever read as a pass; scale it, or send it under its own name= and leave score for the verdict. This is a call of its own because a judge answers after the run it is judging has closed, and a human disagreeing with the judge answers a day later. Re-sending the same name is a correction, not a duplicate:: wai.send_score(“4bf92f3577b34da6”, 0.82, name=“helpfulness”) wai.send_score(scores=[{“traceId”: t, “value”: v} for t, v in graded]) Any other keyword rides along to every measurement sent: label=, pass_at=, max=, description=, direction=, kind=. Returns the gate’s reply — applied, plus unknown (trace ids this account never sent) and rejected when a batch is partly good. Raises when nothing landed at all, because a typo in a trace id is otherwise a silent success.

unpublish

unpublish(dataset_id: str, api_key: str | None = None) -> dict

Take a dataset off the public catalog. The data stays on your account.

update_dataset

update_dataset(
    dataset_id: str,
    purpose: str | None = None,
    mode: str | None = None,
    agent: str | None = None,
    description: str | None = None,
    api_key: str | None = None,
) -> dict

Change what a dataset is for, its mode, agent or description. purpose moves it between the Train, Holdout, Eval and Raw sections of the Training data page. Only the arguments you pass change.

traces

Production traces to a focused coverage grid. Simulation, version two.

dimensions_from_traces

dimensions_from_traces(
    rows: Sequence[dict],
    tools: list[dict],
    policy: str = '',
    broaden: bool = True,
    fault_to_axis: Mapping[str, tuple[str, str]] | None = None,
) -> dict[str, list[str]]

Coverage axes aimed at behaviors seen in rows. Starts from build_dimensions for this agent so every value is one the writer and sandbox understand. The tool axis puts observed failing tools first; broaden=False drops tools the traces never touched (keeping the base specials such as unrelated), so a run spends its budget near the flaws instead of boiling the ocean. Fault and world axes always keep their clean value: contrast needs passing rows too. fault_to_axis maps an observed fault chip to the axis value that reproduces it (FAULT_TO_AXIS by default).

drop_leaky_rows

drop_leaky_rows(
    rows: Sequence[dict],
    sources: Sequence[Any],
    threshold: float = 0.9,
    embedder: Any = 'hash',
    examples: int = 20,
) -> tuple[list[dict], dict[str, Any]]

Kept rows plus the report. Flagged rows are removed, not rewritten.

flaw_rows

flaw_rows(rows: Sequence[dict]) -> list[dict]

Rows with an observed fault or a 0 label: the next round’s traces. The hill-climb loop feeds a round’s failures back into simulate(traces=flaw_rows(evaluated)) so the next batch aims at what the agent still gets wrong.

format_trace_report

format_trace_report(report: dict[str, Any]) -> str

The trace report as a text block, aiming stated in plain words.

infer_harness

infer_harness(rows: Sequence[dict]) -> dict[str, Any]

Draft a harness from observed trace rows. Mechanical, no model. Tool schemas come from what the agent actually sent: every argument key seen for a tool becomes a property, its JSON type read off the observed values, and a key present on every call becomes required. The policy cannot be inferred — exporters do not ship system prompts — so it comes back empty for the caller to fill in. A drafted schema is a starting point to edit, not a spec to trust: it can only describe arguments the traces happened to exercise.

leakage_report

leakage_report(
    generated: Sequence[Any],
    sources: Sequence[Any],
    threshold: float = 0.9,
    embedder: Any = 'hash',
    examples: int = 20,
) -> dict[str, Any]

Near-copy check of generated prompts against source traces. A generated row whose prompt sits at or above threshold cosine similarity to any source prompt is flagged (0.9 by default: the 8-gram exact-overlap test of rlhfbook.com/c/16-evaluation.html with a small paraphrase allowance). Exact matches always flag, whatever the embedder thinks. leaky lists the first examples offenders; n_leaky is the full count.

load_traces

load_traces(source) -> list[dict]

Normalize any supported trace source to the canonical schema above. source is a JSONL path or an iterable of dicts. Rows carrying tool_trace/trace instead of steps, final/output/ response instead of final_text, or only OpenAI-style messages are converted; reward is kept only when it coerces cleanly to 0 or 1, and its absence is fine. Rows that are not dicts or carry neither an ask nor any steps are dropped.

mine_traces

mine_traces(rows: Sequence[dict]) -> dict[str, Any]

What the deployed agent actually did, counted for grid focusing. flaw_rows is any row with an observed fault or a 0 label. Those are the behaviors worth simulating more of.

opening_share

opening_share(rows: Sequence[dict]) -> float

Share of traces whose conversation opens with the assistant. Reads the raw messages field (kept by load_traces); rows without messages count as user-opened. This is the evidence the opening="auto" topology axis resolves against.

simulate_from_traces

simulate_from_traces(
    traces: Sequence[dict],
    agent: Any = None,
    tools: list[dict] | None = None,
    policy: str = '',
    mode: str = 'rl',
    **kwargs: Any,
)

Alias for simulate(agent, traces=...): same grid focus and leakage gate, for callers who start from the traces. With no agent, tools or policy, the tool surface is read from the traces themselves, so handing over graded telemetry is enough to start.

split_pseudo_production

split_pseudo_production(
    rows: Sequence[dict],
    fraction: float = 0.2,
    seed: int = 0,
) -> tuple[list[dict], list[dict]]

Set aside a pseudo-production slice; the rest stays for training. The split is by task, not by row: every row sharing a task_key (the scenario_id, else the prompt) lands on the same side, so the held-out slice is disjoint from the training side in the unit every report groups by, not just prompt-disjoint. Splitting by row is not enough — under mode="rl" with repeats=k each prompt has k rows, and scattering siblings across the two sides trains the student on every prompt it is then evaluated on. Every unique flaw signature (fault name plus behavior shape) sends its task to the production side first, so the held-out slice contains each distinct failure at least once. fraction is still counted in rows, but whole tasks are added, so the slice can overshoot it by up to the size of one task. Deterministic in seed.

tools_from_traces

tools_from_traces(traces: Sequence[dict]) -> list[dict]

The agent’s tool surface, read off the calls the traces contain. Argument names are unioned across every observed call, so a tool called with different arguments in different traces ends up with all of them.

trace_report

trace_report(
    traces,
    tools: list[dict] | None = None,
    policy: str = '',
) -> dict[str, Any]

What these traces contain and what they will aim generation at. Run before simulate(traces=...). With tools (and optionally policy) the report also computes the actual grid emphasis: which axis values move forward in the coverage grid because of these traces. Reward stays optional; ungraded counts are reported, never required. advisory_labels counts rows carrying only a qwen_reward: those labels do steer trace mining, so they are disclosed, not hidden under “ungraded”. dropped counts input rows that carried no usable signal and were discarded by normalization.

​otel

​rows_from_otel

​platform

​agents

​catalog

​cut

​cuts

​datasets

​delete_dataset

​delete_empty_datasets

​format_cuts

​hf_publish

​hf_publish_run

​hf_status

​import_hf

​issue_delegated_credential

​preview

​profile

​publish

​pull

​purge_agent

​push_file

​push_rows

​refresh_delegated_credential

​register_agent

​send_score

​unpublish

​update_dataset

​traces

​dimensions_from_traces

​drop_leaky_rows

​flaw_rows

​format_trace_report

​infer_harness

​leakage_report

​load_traces

​mine_traces

​opening_share

​simulate_from_traces

​split_pseudo_production

​tools_from_traces

​trace_report