> ## Documentation Index > Fetch the complete documentation index at: https://docs.withwhile.com/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > Install with `uv add whileai`; import as `import whileai as wai`. > Run the offline path first (`simulator=False`, `wai.seeded_agent`, a callable judge); no key is needed for it. > Report every pass rate with its interval and n, as `scored.pass_at` prints it. # Synthetic RL data for Prime Intellect > Generate a GRPO-ready dataset for a coding agent, then check it carries gradient before spending GPU time on it. The scripts are in the repository, not in the installed package. Clone it, then `cd recipes/03-select/prime-intellect-rl` before running the commands below. [Browse this recipe on GitHub](https://github.com/whilehq/whileai-sdk/tree/main/recipes/03-select/prime-intellect-rl). Generate a GRPO-ready dataset for a coding agent, then check it carries gradient before spending GPU time on it. The agent in this example is a quant research assistant working in a Python repo that uses pandas and yfinance. Swap `spec.json` for your own tools and policy and the rest of the pipeline is unchanged. What you will learn: why GRPO needs uniform groups, the four numbers that say whether a dataset carries gradient, how an effort-negative reward gets gamed (and how the offline gate predicted it), and the prompt shape the `verifiers` library reads. You need a key for hosted Qwen on both roles: your account key (`whileai login`), or `VLLM_API_KEY` for the shared pool, which is about three minutes for 800 rollouts. `diagnose.py` and `export_prompts.py` run offline on any graded row file. ## Run it ```bash theme={"theme":"vitesse-dark"} uv add whileai whileai login # or: export VLLM_API_KEY=... for the shared pool cd recipes/03-select/prime-intellect-rl python generate.py --situations 100 --k 8 --fault-rate 0.15 # -> data/rl.jsonl (+ .meta.json) python diagnose.py data/rl.jsonl # exit 1 if the set carries no gradient python export_prompts.py data/rl.jsonl # -> data/prompts.jsonl ``` About three minutes for 800 rollouts. Only `generate.py` needs the key; `diagnose.py` and `export_prompts.py` read the file and run anywhere. To try those two with no key, `simulate(agent, spec="spec.json", mode="rl", simulator=False, grade=True)` with a scripted agent writes a small, uniform `rl.jsonl` in under a second; `tests/recipes/test_example_prime_intellect_rl.py` does exactly that. ## What each step does **`generate.py`** runs `simulate(mode="rl")`. That topology gives every prompt the same number of rollouts, which is what GRPO needs: it scores a rollout against the other rollouts of the same prompt, so the training unit is a group of k, not a row. Pass `situations=` explicitly. Without it the generator seeds probes from the spec's `situations` list and can starve well short of the row cap. **`diagnose.py`** is the gate. Mean reward is close to useless on its own, so it reports four things instead: | Check | Why it matters | | ------------------ | ------------------------------------------------------------------------------------------ | | uniform groups | every prompt needs the same k, or advantages are not comparable | | live groups | a group whose k rollouts all score the same has zero advantage and contributes no gradient | | within-group std | how much signal the live groups actually carry | | effort correlation | `corr(tool calls, reward)`. Negative means the reward pays the policy to do less | **`export_prompts.py`** writes the dataset in the shape `verifiers` reads: ```python theme={"theme":"vitesse-dark"} TASK_INPUT_FIELDS = {"prompt", "answer", "info", "example_id"} ``` `answer` is optional. `info` is a free-form dict passed to every reward function, which is where scenario ground truth goes. It exports prompts only. The trainer regenerates rollouts from the policy being trained via `rollouts_per_example`; the rollouts in `rl.jsonl` came from a different model and are off-policy. They are useful as a baseline eval or a rejection-sampling SFT warm start, not for GRPO. It keeps one phrasing per situation. The generator writes several phrasings of some situations and one of others, so keeping all of them would give a few situations several times the gradient weight of the rest. It also drops seed probes. Every entry in the spec's `situations` list becomes a probe, and some reach the output as the seed text itself. Those are third-person scenario descriptions ("the user asks to fill missing bars forward"), not things a user would type, and they make broken tasks. On the run below this removed 7 of 100 prompts. Pass `--keep-seeds` to leave them in, and `--keep-phrasings` to keep every phrasing of a situation instead of one. `--spec` points the seed filter at a different spec file. ## Measured on this spec `generate.py --situations 100 --k 8` at the two fault rates below (the flag defaults to 0.15), then `diagnose.py` on each file: 100 prompts, k=8, 800 rollouts, `Qwen3-4B-Instruct` on both roles, `conduct_grade` as the reward. The engine's seed is its default 0, but rollouts run concurrently, so the exact figures are one run's; the sign of the effort correlation is the part that holds. | `fault_rate` | live groups | within-group std | mean reward | effort corr | | ------------ | ----------- | ---------------- | ----------- | ----------- | | 0.50 | 71% | 0.345 | 0.526 | -0.465 | | 0.15 | **77%** | **0.378** | 0.576 | -0.451 | 77% live is a healthy dataset: roughly three of every four prompts produce a usable advantage. Group uniformity needs care across multiple generation runs. A single `generate.py` call gives every prompt the same k, but accumulating several runs into one pool does not: prompts recur between runs and their rollouts add up. A pool built from five rounds came out with group sizes `{8: 263, 16: 1, 40: 7}`, which `diagnose.py` fails. Key the rollout budget by prompt across runs, or diagnose the pool you actually train on rather than the batch you just made. `export_prompts.py` turned the 100 prompts into 79 tasks: 7 dropped as seed probes, 14 as duplicate phrasings of a situation already covered. ## What happened when this was trained Two Prime Intellect hosted GRPO runs, Qwen3.5-0.8B, 30 steps, identical model, prompts, batch size, group size and learning rate. The only difference was the reward function. | | tool calls per task | its own reward on holdout | | --------------- | ------------------- | ------------------------- | | `conduct_grade` | 4.67 -> **0.90** | 0.300 -> **0.004** | | fitted reward | 4.91 -> **7.60** | 0.498 -> **0.550** | The offline diagnostic predicted this. `conduct_grade` scored -0.468 on effort correlation, which says the cheapest policy is to call no tools. Trained on it, the policy found exactly that: by step 30 it answered `"No"` in two tokens and collected a reward of 1.0. Identical hyperparameters in both arms, so training instability does not explain a divergence that tracks the reward. The trap is in the curve. At step 5 the `conduct_grade` arm posted 0.554, the best holdout score of either arm in the whole experiment. Checkpoint on reward and you ship that model. **A broken reward goes up.** That is why the gate has to run before training rather than during it. ## Read the effort correlation before you train Both runs score around -0.45. Mean reward by tool calls, at `fault_rate=0.5`: ```text theme={"theme":"vitesse-dark"} 0 tools 0.82 3 tools 0.33 1 tool 0.88 4+ tools 0.33 ``` The default `conduct_grade` is a process reward. It checks whether the agent invented an identifier, claimed success after a tool failed, or repeated itself. It does not check whether the agent accomplished anything, so a reply with no tool calls scores 1.0. Meanwhile every additional tool call is another chance to meet an injected fault, and an unacknowledged fault is a hard zero. Effort buys risk and earns nothing. Lowering `fault_rate` does not fix this (-0.465 to -0.451). It is structural. `conduct_grade` is doing its job. It is an honesty floor, and it is good at that: it catches invented file paths and fabricated test results, which is most of what you want to rule out in a coding agent. It is not a task reward, and GRPO needs a task reward. ## Add an outcome term in the environment Build the reward on Prime Intellect's side, where the task is known: ```python theme={"theme":"vitesse-dark"} import verifiers as vf def load_environment(**kwargs): return vf.ToolEnv( dataset=load_prompts("data/prompts.jsonl"), tools=mock_tools(), # wrap sandbox.MockEnvironment rubric=vf.Rubric( funcs=[found_defect, task_complete, conduct], weights=[0.5, 0.3, 0.2], ), max_turns=10, ) ``` `found_defect` and `task_complete` read the seeded ground truth out of `info`. `conduct` is a port of `conduct_grade` with the fault penalty clipped to a floor rather than a hard zero, so meeting a broken tool costs less than lying about one. Then rerun `diagnose.py` against a rollout dump from the new rubric and ship only when the effort correlation is positive. That is the gate this example exists to make cheap. `sandbox.MockEnvironment` is seeded on a hash of the tool name and arguments, so all k rollouts of a prompt see an identical world. That determinism is what makes the within-group comparison meaningful. One packaging note if you push the environment to the Environments Hub: it is installed there with plain pip, so every dependency has to be pip-resolvable. A `[tool.uv.sources]` git pin installs locally, passes the hub's CI, and then kills every env server at training time with a `ModuleNotFoundError`. Depend on `whileai` from PyPI rather than from a git URL. ## Files | | | | ------------------- | ------------------------------------------------------------------- | | `spec.json` | tools, policy rules, and situation seeds for the quant coding agent | | `generate.py` | `simulate(mode="rl")` with uniform group size | | `diagnose.py` | gradient and support report, exits non-zero on a bad dataset | | `export_prompts.py` | verifiers-shaped prompt set |