import whileai.simulations as wai, then wai.name.
SimulationData
SimulationData.grade
grade_llm: the hosted LLM judge (Phi-4,
a different family from the hosted Qwen policy), read from
VLLM_API_KEY. It writes reward and reason onto the rows
in place and returns the judge report (a dict: graded, n0, n1,
backend, judge_version, warnings). llm=True is the same path.
A plain grader= callable scores in place too and returns
nothing. Simulation itself never invokes this method by default.
judge= is the contract path: any callable honoring the judge
contract (judge(row) -> \{"reward": 0 or 1, "reason": str, "markers": \{name: value\}\}; a bare number works too). The
contract and its failure modes are written out in full in
whileai.simulations.score.judging — note the score.,
there is no whileai.simulations.judging. It returns a
ScoredData of copies — trajectories here stay unmodified, judge
errors are marked per-row instead of coerced to 0 — and its output
feeds export_training and simulate(traces=...) directly.
version= names the judge’s version (model, rubric hash) and is
recorded on every scored row; the hosted grader stamps its own.
Every path then checks the judge against the rows’ human labels
(attach_labels(kind="human")) and stamps the summary on each
graded row’s judge_meta["trust"]. trust="warn" (default)
logs one line when the check failed or no labels exist,
"require" raises instead, "off" skips it.
SimulationData.grade_llm
WHILEAI_JUDGE is set), never the policy model.
use_privileged shows the judge each row’s privileged block
(principle, reference, hidden state) the agent never saw. trust
is the judge check against human labels: see grade.
payload_chars caps the evidence the judge reads per row and
max_tokens its reply (defaults JUDGE_PAYLOAD_CHARS and
JUDGE_MAX_TOKENS in defaults.py); both land in
judge_meta.
SimulationData.leak_report
privileged block? Reads the
trajectories, which still carry the block; rows() is scrubbed
and would check nothing. Same report as leak_report.
SimulationData.llm_grade
SimulationData.push
purpose is the section it lands in on the Datasets page
("train" by default; "holdout" or "eval").
holdout=0.2 keeps a fifth of the tasks (by scenario_id) out
of the training set and pushes them as a second, linked dataset
with purpose "holdout"; the entry carries it as ["holdout"].
The simulation mode is recorded on both.
api_key defaults to the WHILEAI_API_KEY env var, then the
key saved by whileai login. Pass parent (a ds_...
id) when this run iterates on an existing dataset, so lineage shows
on the platform. publish=True with an agent name also puts it
on the public catalog at zeroproofai.com/datasets as a card. Returns
the registry entry with datasetId.
gate=True runs publish_gate first: every graded row gets a
calibration stamp (per-task pass rate, k, producing policy),
and an RL-shaped run that is ungraded or has no mixed group is
refused with PublishGateError. The gate report is returned as
entry["gate"]. gate=False uploads rows as they are.
endorsed names what the reward should track (feature-name
substrings, e.g. "tool:lookup_order") for the gate’s
hack_scan; strict_hacks=True refuses a set whose reward
is best explained by something else.
SimulationData.rank
reward untouched.
Writes quality, quality_reason, quality_scores on each
trajectory and rewrites the saved JSONL, or path if you pass one.
SimulationData.report
data.coverage).
SimulationData.select
select_for_sft: one of
each distinct way of being right before any repeats, junk and
duplicate prompts dropped. Requires graded rows — grade in-loop
(grade=True, grader=) or afterwards with grade().
The selection report lands in search["selection"].
SimulationData.training_set
select() picks diverse pass-labeled rows, export_training
writes them as chat JSONL with this run’s system prompt and tools
and the tool-call round-trip gate. Returns the export report with
the selection report attached; pass output to write the file.
Raw simulation rows are not the training artifact — this is.
conversation
grade
source is a SimulationData, a JSONL path, or a row list.
Writes reward 0 or 1 and a one-sentence reason. Keeps the
previous score as qwen_reward when present. Does not run during
simulate(). Search does not read reward. limit grades
that many rows then stops. Hosted Qwen reads VLLM_API_KEY.
For a path or row list, pass policy= and tools= so the judge
sees the agent’s rules; a SimulationData supplies its own.
trust is the judge check against human labels: see
SimulationData.grade.
grade_llm
source is a SimulationData, a JSONL path, or a row list.
Writes reward 0 or 1 and a one-sentence reason. Keeps the
previous score as qwen_reward when present. Does not run during
simulate(). Search does not read reward. limit grades
that many rows then stops. Hosted Qwen reads VLLM_API_KEY.
For a path or row list, pass policy= and tools= so the judge
sees the agent’s rules; a SimulationData supplies its own.
trust is the judge check against human labels: see
SimulationData.grade.
llm_grade
rank
source is a JSONL path, a row list,
or a SimulationData. Does not change simulate() or reward.