The scripts are in the repository, not in the installed package. Clone it,
then
cd recipes/03-select/prime-intellect-rl before running the commands below. Browse this recipe on GitHub.spec.json for your own tools and policy and
the rest of the pipeline is unchanged.
What you will learn: why GRPO needs uniform groups, the four numbers that say
whether a dataset carries gradient, how an effort-negative reward gets gamed
(and how the offline gate predicted it), and the prompt shape the verifiers
library reads. You need a key for hosted Qwen on both roles: your account
key (whileai login), or VLLM_API_KEY for the shared pool, which is
about three minutes for 800 rollouts. diagnose.py and export_prompts.py run
offline on any graded row file.
Run it
generate.py needs the key;
diagnose.py and export_prompts.py read the file and run anywhere. To
try those two with no key, simulate(agent, spec="spec.json", mode="rl", simulator=False, grade=True) with a scripted agent writes a small, uniform
rl.jsonl in under a second; tests/recipes/test_example_prime_intellect_rl.py
does exactly that.
What each step does
generate.py runs simulate(mode="rl"). That topology gives every prompt the
same number of rollouts, which is what GRPO needs: it scores a rollout against the
other rollouts of the same prompt, so the training unit is a group of k, not a row.
Pass situations= explicitly. Without it the generator seeds probes from the
spec’s situations list and can starve well short of the row cap.
diagnose.py is the gate. Mean reward is close to useless on its own, so it
reports four things instead:
export_prompts.py writes the dataset in the shape verifiers reads:
answer is optional. info is a free-form dict passed to every reward function,
which is where scenario ground truth goes.
It exports prompts only. The trainer regenerates rollouts from the policy being
trained via rollouts_per_example; the rollouts in rl.jsonl came from a
different model and are off-policy. They are useful as a baseline eval or a
rejection-sampling SFT warm start, not for GRPO.
It keeps one phrasing per situation. The generator writes several phrasings of
some situations and one of others, so keeping all of them would give a few
situations several times the gradient weight of the rest.
It also drops seed probes. Every entry in the spec’s situations list becomes a
probe, and some reach the output as the seed text itself. Those are third-person
scenario descriptions (“the user asks to fill missing bars forward”), not things a
user would type, and they make broken tasks. On the run below this removed 7 of
100 prompts. Pass --keep-seeds to leave them in, and --keep-phrasings to
keep every phrasing of a situation instead of one. --spec points the seed
filter at a different spec file.
Measured on this spec
generate.py --situations 100 --k 8 at the two fault rates below (the
flag defaults to 0.15), then diagnose.py on each file: 100 prompts, k=8,
800 rollouts, Qwen3-4B-Instruct on both roles, conduct_grade as the
reward. The engine’s seed is its default 0, but rollouts run concurrently,
so the exact figures are one run’s; the sign of the effort correlation is
the part that holds.
77% live is a healthy dataset: roughly three of every four prompts produce a
usable advantage.
Group uniformity needs care across multiple generation runs. A single
generate.py call gives every prompt the same k, but accumulating several runs
into one pool does not: prompts recur between runs and their rollouts add up. A
pool built from five rounds came out with group sizes {8: 263, 16: 1, 40: 7},
which diagnose.py fails. Key the rollout budget by prompt across runs, or
diagnose the pool you actually train on rather than the batch you just made.
export_prompts.py turned the 100 prompts into 79 tasks: 7 dropped as seed
probes, 14 as duplicate phrasings of a situation already covered.
What happened when this was trained
Two Prime Intellect hosted GRPO runs, Qwen3.5-0.8B, 30 steps, identical model, prompts, batch size, group size and learning rate. The only difference was the reward function.
The offline diagnostic predicted this.
conduct_grade scored -0.468 on effort
correlation, which says the cheapest policy is to call no tools. Trained on it,
the policy found exactly that: by step 30 it answered "No" in two tokens and
collected a reward of 1.0.
Identical hyperparameters in both arms, so training instability does not explain
a divergence that tracks the reward.
The trap is in the curve. At step 5 the conduct_grade arm posted 0.554, the
best holdout score of either arm in the whole experiment. Checkpoint on reward
and you ship that model. A broken reward goes up. That is why the gate has to
run before training rather than during it.
Read the effort correlation before you train
Both runs score around -0.45. Mean reward by tool calls, atfault_rate=0.5:
conduct_grade is a process reward. It checks whether the agent
invented an identifier, claimed success after a tool failed, or repeated itself.
It does not check whether the agent accomplished anything, so a reply with no tool
calls scores 1.0. Meanwhile every additional tool call is another chance to meet an
injected fault, and an unacknowledged fault is a hard zero. Effort buys risk and
earns nothing.
Lowering fault_rate does not fix this (-0.465 to -0.451). It is structural.
conduct_grade is doing its job. It is an honesty floor, and it is good at that:
it catches invented file paths and fabricated test results, which is most of what
you want to rule out in a coding agent. It is not a task reward, and GRPO needs a
task reward.
Add an outcome term in the environment
Build the reward on Prime Intellect’s side, where the task is known:found_defect and task_complete read the seeded ground truth out of info.
conduct is a port of conduct_grade with the fault penalty clipped to a floor
rather than a hard zero, so meeting a broken tool costs less than lying about one.
Then rerun diagnose.py against a rollout dump from the new rubric and ship only
when the effort correlation is positive. That is the gate this example exists to
make cheap.
sandbox.MockEnvironment is seeded on a hash of the tool name and arguments, so
all k rollouts of a prompt see an identical world. That determinism is what makes
the within-group comparison meaningful.
One packaging note if you push the environment to the Environments Hub: it is
installed there with plain pip, so every dependency has to be pip-resolvable. A
[tool.uv.sources] git pin installs locally, passes the hub’s CI, and then kills
every env server at training time with a ModuleNotFoundError. Depend on
whileai from PyPI rather than from a git URL.