simulator=False writes situations
from templates, seeded_agent stands in for an agent, and a lambda can
be the judge, all offline. A key adds the hosted situation writer, the
hosted judge, dataset storage, training and serving. See
install.
How do I know it got better? Every run ends with pass@1 per task
category, before and after, on a held-out set with a 95% confidence
interval. If the interval includes zero, nothing ships.
Which models can it train? Open models you own at the end: Qwen,
Llama, Nemotron and other open-weight families. Training runs SFT, DPO or
GRPO on hosted GPUs, and the result is served from your own endpoint. See
the platform reference.
Where do the rollouts come from? From your agent’s own tools and
policy. A separate model plays the customer, a mock world answers the tool
calls, and every conversation is scored against a written rubric before it
is kept. See how it works.
Is the judge another LLM? Yes, when you do not pass a program. So it
is measured against gold labels, probed with known hacks, versioned by
rubric hash, and drawn from a different model family than the policy. See
the engine.
What happened to zeroproof? Same package, renamed in 0.51.
pip install zeroproof and import zeroproof.simulations still work and
give you whileai.