Skip to main content
The scripts are in the repository, not in the installed package. Clone it, then cd recipes/community/hosted-grpo-vs-sft before running the commands below. Browse this recipe on GitHub.
Seat: an open-model fine-tuner who lives in TRL and Hugging Face, here for clean exports, a LoRA run, a hosted result and a cost line. Question: the customer-simulation ledger (#31) has said four times that nobody has run method="grpo" on the hosted path. So: run hosted SFT and hosted GRPO on the same 84 rows, measure both against the same base on a pinned holdout, and see whether the adapter can be pulled back out into PEFT form. The answer, and it is not about GRPO. Neither method moved the target on 19 tasks (nowhere near enough tasks — push_rows said so before I trained). What the run actually found is that the hosted trainer silently disagrees with itself about which rows to train on, in two different directions, and neither one tells you: And the SDK already knows this is dangerous. The local exporter, on the identical rows, refuses to stay quiet:
wai.export_training prints that. wai.train(method="sft") does not, and then trains on those 72 rows. The warning predicted my result: the SFT adapter half-stopped using its tools at all, called_lookup 0.987 → 0.487, −0.500 [−0.697, −0.303], p=0.0025. It learned the failure, exactly as the other half of the product said it would. What you will learn: what the hosted sft / grpo / dpo methods actually consume, why profile()["mixed_tasks"] is the number that predicts whether a GRPO run will do anything, why a guard marker belongs in must_not_regress even when it reads 1.000, and that wai.hf_publish_run works in 20 seconds. You need WHILEAI_API_KEY; --dry-run needs nothing. Two hosted runs, under $1 of GPU (see Cost, and note the SDK gave me no way to compute that).

​
Run it

​
The rule

One deterministic marker over tool calls only — no prose is parsed, no dollar figure is matched. Three separate ledger entries found their headline was a defect in a grader that read the reply text, so this one does not read reply text at all. The guard reads 1.000 on every arm and is still worth carrying: it is satisfiable by inaction, so it can only ever regress, which is precisely what a guard is for. The target is the one with headroom (base 0.250). python run.py --dry-run pins all six rows of this table offline, including the two degenerate cases (“asked for the id” and “lookup only” both fail the target). Denominator, stated plainly. The task grid contains prompts that legitimately have no order id, where refusing to refund is correct. They sit in the denominator, so the level of resolved_with_lookup is not an accuracy. The task set is pinned and identical across all three arms, so the paired delta is like for like — which is all this recipe claims.

​
What I actually ran

  • 160 rollouts from wai.simulate (offline template writer, no model key), 40 prompts × 4 repeats, agent = served qwen3-4b-think through wai.local_model, 108 s
  • split_pseudo_production(fraction=0.35) → 84 train / 76 holdout
  • Pushed the same 84 train rows twice, under two ids. wai.train on a dataset that is already training answers with that run instead of starting a second one, so two methods over identical rows need two dataset ids.
  • Both runs: base_model="Qwen/Qwen3-4B", lora_r=16, seed=11, same holdout set
  • All three arms evaluated through one entry point (wai.local_model, thinking=False, temperature=0.8), on the same 76 pinned holdout rows / 19 tasks, k=4 — the same-entry-point discipline from same-entrypoint-before-after

​
Results

The power warning came first, unprompted, from push_rows:
Believe that before you read anything below. 19 tasks resolves nothing subtle. Paired, by task, 95% bootstrap interval from wai.delta_report, n_paired = 19: delta_report put both tool markers in slipped for the SFT arm and left regressions empty, since the target did not regress. The interesting number is in slipped, not in the verdict — worth knowing before you read only the headline. Neither adapter beat the base on the target. Say that plainly. The SFT arm is a measured, significant behavioral regression on tool use; the GRPO arm is a +0.053 that 19 tasks cannot resolve from zero.

​
Why GRPO did nothing: 6 rows

GRPO needs a group with both a pass and a fail for the advantage to be non-zero, and wai.profile(dataset_id) had already computed exactly how many of my tasks qualified:
mixed_tasks: 6 is the number that predicted this run would be a no-op, and it is available before you spend the GPU. Nothing in wai.train looked at it or warned. 20 steps over 6 prompts ran 3.3 epochs and ended at reward_std 0.0, grad_norm 0.0 — a collapsed run that still reports status: done. The hosted GRPO reward is fixed (config.reward = "reference first action", config.gold = "judge"); you do not pass a reward function on this path. That is a real design choice and it is not documented at the call site. The run’s hack monitor did fire, on its own:
On a run whose reward never moved, I read that as the monitor finding structure in noise rather than a real exploit — but it cost nothing and it is the right instinct.

​
The export half: this worked

20 seconds, 148 MB, 10 files, private repo, commit tagged zp-run_7d45f7e7f724464a: jaweiss2305/refund-lookup-grpo-qwen3-4b. wai.hf_status() told me the account was connected and which namespaces I could write to before I spent anything, which is the check I wanted and did not expect to find. I did not verify the repo loads under peft.PeftModel.from_pretrained — the repo is private and this sandbox has no HF_TOKEN. 148 MB is larger than an r=16 LoRA on a 4B needs, so check what the 10 files are before assuming it is adapter-only. wai.export_training(rows, path, format=...) is the local half and is good: OpenAI wire format by default (arguments as a JSON string), format="trl" for the dicts a chat-template trainer wants, mask_mode, strip_think, and the reward warning quoted at the top of this README.

​
Cost

The dollar column is mine, not the product’s. Six ledger entries have now asked for a cost number. run.summary gives seconds and gpu — which is genuinely more than it used to — but there is no published rate anywhere in the SDK, wai.models(), or withwhile.com, so the right-hand column is me multiplying by a GPU price I brought from outside. Do not cite it as a While number.

​
What I would do next

  1. Size the set before training, not after. profile() gives mixed_tasks and support; a GRPO run wants those to be large. My support: 0.238 was the tell.
  2. Filter the SFT set yourself. Until train(method="sft") matches its docstring, push only passing rows (scored.passes(), or filter reward == 1.0) into the set you hand it. My 72 failure rows went straight into the adapter.
  3. Get more tasks. 19 is not an experiment. holdout_size says 352 for a 5% gain; the offline template writer will produce them and they cost no model key.
  4. Re-run GRPO on a set built for it — tasks deliberately sampled to be mixed — and see whether the hosted path can move anything at all. That question is still open: this run did not test GRPO, it tested a 6-row GRPO run.

​
Issues this run filed

  • #396 train(method="sft") trains on failing rows; export_training warns and train does not
  • #397 train(method="grpo") silently used 6 of 84 rows
  • #398 dataset_report("ds_...") returns an all-zero report instead of raising
  • #399 no cost number anywhere
  • #400 simulate() prints nothing until it returns

​
Run ids

The three datasets were deleted after the run (scratch); the runs and the two served models are kept, and the serving endpoint scales to zero on its own.
Last modified on September 19, 2026