Quickstart

No key, no network. seeded_agent is a stand-in agent. It answers honestly most of the time and, on a labeled fraction of rollouts, does one thing wrong on purpose: hedges, flatters, or claims success after a tool failed. Each row records what it did in seeded, so you can check that your judge catches exactly those rows before you trust it on real ones.

Describe the tools

Tools go in OpenAI function-calling shape. One is enough.

import whileai.simulations as wai

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "get_order",
            "description": "Look up an order by id.",
            "parameters": {
                "type": "object",
                "properties": {"order_id": {"type": "string"}},
                "required": ["order_id"],
            },
        },
    }
]

Simulate

simulator=False writes the customers from templates, so no model is called. mode="rl" with repeats=4 plays every ask four times, which is what pass@k needs.

data = wai.simulate(
    wai.seeded_agent(TOOLS),
    tools=TOOLS,
    system_prompt="Help customers with orders.",
    simulator=False,  # no model
    mode="rl",
    repeats=4,
    repeat_policy="fixed",
    budget=64,
)

Grade and read the number

Any callable that takes a row and returns a reward is a judge. This one uses the label the stand-in agent left behind.

scored = data.grade(judge=lambda row: {"reward": int(not row["seeded"])})
print(scored.pass_at)

pass@1 0.67 [0.55..0.78] | pass^4 (pass_pow_k) 0.19 [0.00..0.38] | pass@4 1.00 [1.00..1.00] | headroom 0.33 (16 groups, k=4)

pass@1 is the pass rate over tasks with a bootstrap interval. pass^4 is how often all four rollouts of a task pass. Headroom is pass@4 minus pass@1, the gap an RL update could close.

Connect your agent

A callable, an OpenAI-compatible endpoint, a model spec string, or the hosted model. Plus traces to aim the run.

Evals

A pass rate with an interval, a table of where the agent fails, and a CI check that turns red when it gets worse.

Simulations

How the engine picks situations, plays the customer, and breaks the tools on purpose.

The five calls

simulate, grade, optimize, export, train: the run in order.

Get started

Guides

Concepts

Reference

API

Next

Connect your agent

Evals

Simulations

The five calls

​Next

Connect your agent

Evals

Simulations

The five calls

Next