Skip to main content
The scripts are in the repository, not in the installed package. Clone it, then cd recipes/01-simulate/bring-your-own-agent before running the commands below. Browse this recipe on GitHub.
Three things a first run with your own agent needs: the callable contract, what the run says when the agent is broken, and how an eval score is kept out of the training reward. Offline, no key, seconds.

​
The contract

The engine calls your function once per rollout with the situation the simulator wrote, and expects the tool calls it made and what it said:
simulator=False uses the built-in template writer so no model key is needed; the situations are less varied than a model writes, which is fine for wiring up an agent and a judge. Any extra keys on the dict stay on the row. Inside the callable, current_rollout.rollout_index says which repeat this is, if the agent needs to know. repeat_policy="fixed" asks for all four repeats up front. The mode="rl" default is "successive": two probe rollouts per ask, and the remaining repeats only on asks whose graded probes disagree. That is the right economy for a graded run, but an ungraded one never splits, so it would stop at two repeats per ask and the example’s every-third-repeat agent would never misbehave.

​
When the agent is the problem

A callable that raises, or returns an OpenAI-style message instead of the contract, used to produce a zero-row run that blamed the situation writer. Now the run says so:
The error count is the number of rollouts the engine tried before giving up, a little over budget=8; it moves by a few between runs. data.stopped_because == "agent_failed" when no row survived, data.search["agent_errors"] is the count, data.search["first_agent_error"] carries the exception type and message, and "agent_errors" lands in data.degraded whenever any rollout was lost this way, even on a run that still produced rows.

​
An eval score is not a reward

run_judge(rows, judge) and evaluate(rows, judge) produce the same row shape and the same numbers. The difference is provenance: evaluate stamps lineage.source == "eval" because it is meant for a held-out set. Training on those rows makes the scorer you report the reward you optimised against.
select_for_rl, select_for_sft and build_preference_pairs each report eval_sourced and warn when it is non-zero. Nothing is dropped; the count is the guard. Grade the training set with run_judge (or data.grade(judge=...)), and keep evaluate for the rows the model never sees.
Last modified on September 19, 2026