> ## Documentation Index
> Fetch the complete documentation index at: https://docs.withwhile.com/llms.txt
> Use this file to discover all available pages before exploring further.

# whileai

> Training and eval data for agents that call tools. Give it an agent, get graded conversations back, train on the ones that carry signal, and prove the gain on a held-out set.

`whileai` makes training and eval data for agents that call tools. Give it
an agent, or just the agent's tools and system prompt. It writes the
situations the agent might meet, runs the agent through them against a
fake world that fails on purpose, and hands back every conversation as a
row. You grade the rows with your own judge or a verifier. The package
then does the bookkeeping that is easy to skip and expensive to get wrong:
pass rates with intervals, difficulty bands for RL, a check that your judge
agrees with people, decontamination against your eval set, and a scan for
rewards the policy can game.

```bash theme={null}
pip install whileai
```

<CardGroup cols={2}>
  <Card title="Quickstart" icon="bolt" href="/get-started/quickstart">
    Sixty seconds, offline, no key. A stand-in agent, a one-line judge,
    pass\@1 with an interval.
  </Card>

  <Card title="Connect your agent" icon="plug" href="/get-started/connect-your-agent">
    A callable, an endpoint, a model spec string, or the hosted model.
    Traces to aim the run.
  </Card>

  <Card title="Evals" icon="flask" href="/evals">
    You only want a number that says how often the agent does the job,
    and a test that goes red when it gets worse.
  </Card>

  <Card title="Train" icon="dumbbell" href="/reference/five-calls">
    Grade, keep the rows that carry signal, export to your trainer or
    ours, measure the delta on held-out tasks.
  </Card>
</CardGroup>

## The loop

| Step               | Call                                                                 | What it computes                                                                                                            |
| ------------------ | -------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| Simulate           | `simulate(agent, tools=, system_prompt=, mode="rl", repeats=k)`      | A covering array over tools, world state and user stance. k rollouts per prompt. Scheduled tool faults.                     |
| Grade              | `data.grade(judge=)`, `verify.MathEqual`, `verify.CodeExec`          | A reward per rollout under one contract. Verifiable rewards where a program can check.                                      |
| Validate the judge | `judge_trust`, `judge_probes`                                        | Agreement and Cohen's kappa against human gold. Length bias. Exploit probes.                                                |
| Measure            | `pass_at`, `delta_report`, `eval_variance`, `holdout_size`           | pass\@1, pass^k, pass\@k with bootstrap intervals over tasks. A paired delta with a permutation p-value. Noise band. Power. |
| Select             | `optimize(mode="rl"\|"sft")`, `build_preference_pairs`, `curriculum` | The 20 to 80% difficulty band. Unanimous groups dropped. Rejection sampling. Length-matched pairs.                          |
| Guard              | `decontaminate`, `hack_scan`, `trace_markers`, `HackMonitor`         | Overlap with the eval set. Reward-feature correlation within task against a shuffle floor. Trajectory lies.                 |
| Train and export   | `export_dataset`, `export_environment`, `train`, `serve`             | Loss masks. A `verifiers` environment for GRPO. Hosted LoRA SFT, GRPO, DPO, RM.                                             |

Where each method comes from, with numbered references, is on
[the engine](/concepts/engine). The calls in the order a run happens are
on [the five calls](/reference/five-calls). Every public call, with its
signature and docstring, is under API in the sidebar; those pages are
generated from the package on each release, so they cannot drift.

## Guides

<CardGroup cols={3}>
  <Card title="Simulations" href="/simulations">
    How the engine thinks and why.
  </Card>

  <Card title="Character training" href="/character-training">
    A stable way of talking, in the weights.
  </Card>

  <Card title="Reward hacking" href="/reward-hacking">
    The gap between the training reward and the eval you care about.
  </Card>

  <Card title="Safety evals" href="/safety-evals">
    Private data, actions, sends: can they be turned against the owner?
  </Card>

  <Card title="Platform" href="/reference/platform">
    Sign in, store datasets, train on hosted GPUs, serve the adapter.
  </Card>

  <Card title="Recipes" href="https://github.com/whilehq/whileai-sdk/tree/main/recipes">
    Runnable, one folder each, offline where the method allows.
  </Card>
</CardGroup>