Skip to main content
A rollout has four pieces. Three train the model. One evaluates it.
The test is not training data. It is the only piece the model never sees, which is why a score on it means something.

​
The loop

1

Watch

See where the agent fails at work. Live traffic is read into a map of what the agent handles well and what it does not.
2

Simulate

Run it through realistic scenarios. The weak spots become thousands of rollouts, with tools that break on purpose and customers who do not cooperate.
3

Train

Every rollout gets a reward, and the ones that pass become SFT, DPO or GRPO data for the next version of the model.
4

Prove

Evaluate on a held-out set. A paired before and after per task category, with a 95% confidence interval.
Then it starts again. The next day’s traffic is the next training set.

​
Five steps inside one run

1

Pick a prompt

Each prompt combines a task category, a tool condition and a customer stance. Sometimes the tool is broken on purpose.
2

Track coverage

Every prompt lands in a square on a grid of situations. A few per square counts as covered. Generation stops when the squares stop filling.
3

Focus on what works

There are several ways to come up with the next prompt. The ones that keep finding new behavior get more of the budget.
4

Run the conversation

A separate model plays the customer. A mock world answers every tool call. Two results come out: a score to train on and a test kept out of training.
5

Cut it into training data

Good replies become examples. Good and bad replies to the same prompt become pairs. The tests go in a set of their own.

​
What goes in

​
What comes out

​
Proving it

After training, the new model answers the test set. So does the old one. The report is a pass@1 per task category, before and after, so a gain on refunds cannot hide a loss on cancellations. Nothing counts as better until the held-out interval excludes zero. The first public run of this method is written up in We trained a support agent to hand customers off at the right time: a 15-dollar simulated set took a telecom support agent, one that runs a phone company’s support chat with real account tools, from 5% to 30% on a public benchmark. The mechanics behind each step, with references, are on the engine. The calls that run them are in the reference.