Skip to main content
The scripts are in the repository, not in the installed package. Clone it, then cd recipes/04-train/text-to-sql before running the commands below. Browse this recipe on GitHub.
A question about a database in, one SQL query out, and a reward that is a program: run the query, compare the result set to the gold query’s result. No judge. This recipe builds the task set for a schema, benchmarks any model on it, trains a 4B model with GRPO against that reward on Modal, and measures each round on the same held-out tasks, so the curve is paired, has an interval, and cannot be gamed by a wordier answer. The schema shipped here is a small online store (8 tables, 300 orders, seeded from gen_seed.py). Swap in yours: Bring your own schema.

​
What you get

​
Numbers so far (81 held-out tasks, 4 samples each, temperature 0.7)

Two things to read off that table before training anything. pass@4 minus pass@1 is the headroom a grouped update can amplify: 0.06 with thinking off, 0.12 with it on. And where the model loses: multi-table joins, self-joins, date buckets, derived metrics (the archetype table build.py prints). Training the thinking-off model went nowhere, as the headroom predicted: hosted SFT on the 336 gold demonstrations 0.58 -> 0.63 and GRPO with the execution reward 0.58 -> 0.59, both intervals covering zero. The hill climb is in thinking mode; rounds and their numbers are at the bottom.

​
Run it

Needs: Python 3.11+, uv add "whileai>=0.47" "psycopg[binary]" openai anthropic, a Postgres you can create a database on, WHILEAI_API_KEY from withwhile.com/platform (the hosted Qwen3-4B endpoint, the datasets page and the training page), and a Modal account for the RL step. 1. The database. Any Postgres 16 works; the scripts read T2S_PG_DSN (default postgresql://postgres@127.0.0.1:5499/shop).
2. Benchmark the base model. Four samples per held-out task; rows land in raw/<model>.jsonl and the run resumes if interrupted.
Rollouts go through wai.simulate(agent, system_prompt=..., tasks=..., repeats=k): the SDK replays the task prompts on the agent and the gold SQL is attached to the rows afterwards. Thinking on for the base model: serve it under a name and sample that (wai.serve("qwen3-4b-think", base_model="Qwen/Qwen3-4B"), then --hosted qwen3-4b-think). Any other model: --agent openai:<model> with OPENAI_BASE_URL, or add a callable to MODELS. 3. Train, round one. GRPO on Qwen3-4B, LoRA rank 16, 8 samples per prompt, the execution reward (1.0 on a result match, 0.1 when the query runs but is wrong, 0 otherwise). Postgres is installed in the image and seeded in the container, so the reward needs nothing from your machine.
About 65 s a step in thinking mode on an H100 (completions average 900 tokens), so 100 steps is under two hours and roughly $8. --skip-eval skips the slow in-container before/after sampling; the measurement comes from the served adapter in the next step, through vLLM, in minutes. --spawn submits the call and returns, so nothing depends on your laptop staying connected (a network drop cancelled a 3 h run at step 49 without it). The run shows on your training page as it goes (reward, KL, completion length); the adapter and summary.json land on the whileai-train-runs volume under the run id. 4. Serve and measure. The adapter is saved on the whileai-train-runs volume under the run id, which is what wai.serve hosts.
delta.py prints the paired before/after with a 95% interval, by difficulty and by archetype, and attaches it to the run page. 5. Round two. Start from round one’s adapter, measure the same way.

​
Runs in parallel

Every run is its own Modal container with its own Postgres and its own directory on the volume, keyed by run id, so launch as many as you like at once and compare on the same holdout:
Then serve each, rollout.py --hosted <name>, and one build.py prints them side by side. Knobs: --learning-rate, --beta, --steps, --num-generations, --loss-type (bnpo, grpo, dr_grpo), --max-completion-length, --lora-rank.

​
Why the tasks are authored, not simulated

wai.simulate writes situations, rollouts and world state; it never writes an answer key, so a verifiable task set is rows you bring that carry privileged.reference (the SDK README says the same under Verifiers). For SQL the reference has to be a query that is exactly right on the data, and the question has to be unambiguous about which rows count and what to return, or an exact-match verifier punishes valid readings. author.py therefore has a teacher write the question and the gold together, executes the gold twice, and keeps it only when it returns 1-50 stable rows. Once the tasks exist, everything downstream is the SDK: simulate(tasks=...) for rollouts, data.grade(judge=SQLExec()), pass_at, optimize, push_rows, training_run, serve, delta_report.

​
Bring your own schema

  1. Replace schema.sql (DDL, comments welcome: the model reads them) and seed.sql (data; a few hundred rows is enough, the point is that the gold queries have answers).
  2. Rewrite NOTES in schema_prompt.py: the formulas and NULL meanings a new analyst would need. Run python schema_prompt.py to refresh prompt.txt.
  3. Load the database (step 1 above) and write tasks: python author.py --rounds 2 (Claude Sonnet 5; ANTHROPIC_API_KEY, or AWS credentials for Bedrock; ~15 min and a few dollars for ~450 tasks). Every gold query is executed twice and must return 1-50 rows; near duplicates are dropped. Read a sample of tasks.jsonl: the questions must be unambiguous about which rows count and what to return, because the verifier is exact.
  4. Steps 2-5 above, unchanged.
The verifier (sql_verifier.py: SQLExec, the same module the trainer mounts) compares result sets as multisets, floats rounded to 2 places, text case-folded, columns in any order, and in order only when the gold query has ORDER BY. It reads the gold from privileged.reference, which the training export never projects, so the answer key cannot leak into a training file.

​
What we learned building it

  • Headroom first. pass@4 - pass@1 says whether RL has anything to amplify. At 0.06 nothing moved in 300 steps; at 0.12 it does.
  • Thinking off is the wrong regime for this model. 0.58 vs 0.69 for the same weights.
  • gradient_checkpointing=True broke generation. With TRL 0.19.1 and transformers 4.54 on Qwen3-4B, checkpointing makes TRL generate without a KV cache and every training completion was random tokens with reward 0, while a plain model.generate was fine. It is off here; memory is handled by micro-batches (--accum 4 = 8 samples in four batches of 2) and an H100. An L40S OOMs at this prompt length.
  • Sampling at 0.7 gives half-duplicate groups. optimize(mode="rl") drops them; the RL set from 8 samples on 336 prompts was 144 rows in 38 groups, pool_exhausted in the scan. Train on the prompts, not the set.
  • Code-fenced replies read as truncated to looks_finished before 0.46 (whileai-sdk#212, fixed in 0.46); build.py carries the same patch so it also runs on 0.44.
  • Thinking models need a reply budget. simulate(agent_max_tokens=4096, timeout=300) (whileai >= 0.47); on the default 2048-token cap and 60 s timeout the base lost 8% of replies mid-thought and 4 of 81 tasks.
  • The adapter you serve is not always the adapter you trained. A Qwen3.5 LoRA saved by PEFT on transformers 5 loads in vLLM 0.29 without a warning and changes nothing (key layout, whilehq/whileai-sdk#588). Before reading any served-checkpoint number, score the adapter on 100 prompts it trained on and compare with the trainer’s own reward on them; a gap of 0.3 is a serving bug, not a generalization gap.
  • Grade the turn you asked for. simulate(max_turns=1) still wrote a second user turn when the reply held a ”?” and the grader scored that reply; every Qwen3.5 checkpoint read 7 points low and the 4B 21 points low until the rows were cut at the first reply (whilehq/whileai-sdk#586). Count assistant turns per row before reading any table.
  • Re-cut the band every round. A prompt band chosen on the base is solved by the end of round 1; round 2 on the same band ran at reward 0.95 with 60-88% zero-variance groups and moved nothing. Read frac_reward_zero_std in the first steps; above 0.5, stop and re-cut.
  • A reply budget needs a matching call timeout. Qwen3.5-9B writes up to 4,096 tokens of plain-text reasoning; at 32 concurrent requests on one L40S that is about 28 tokens a second per request, so the long replies took longer than the 300 s default and the SDK re-rolled each one up to repeats times. Sampling 601 prompts ran 6.5 hours and never finished. rollout.py --timeout 900 --concurrency 16 finishes; the rule is timeout
    = max_tokens / per-request tokens per second. Since whilehq/whileai-sdk#470 the SDK’s default is that rule at 4 tokens a second, max(300, agent_max_tokens / 4) (1,024 s at 4,096 tokens), so leaving timeout= unset no longer re-rolls a reply for taking the time it was allowed.
  • Pinned prompts do not need the situation writer. With --agent "vllm:..." the engine drafts scene briefs and result shapes with the same model even when every prompt is pinned (degraded says scene_brief_unavailable, the run says same_model), and a model that writes 2,000-4,000 tokens a reply spends a large share of the server on it. rollout.py --no-writer passes simulator=False; a four-prompt probe went 177 s to 133 s. Sample long runs in chunks (--limit 75, then 150, …): rows already on disk are skipped, so a killed run resumes, and each chunk prints its rate (whilehq/whileai-sdk#470).
  • A long run is no longer a black box. The first 2,400-row pass ran four hours with nothing on stdout, no row on disk until it returned, and every row lost on a kill; the cause (long replies timing out and re-rolling up to repeats times) was only visible in the server log. whilehq/whileai-sdk#470 put three things in simulate(): a progress line on the whileai.simulations logger (stderr when nothing listens) every 10 events or 10 s that counts rows landed, re-rolled and lost by reason, with the same dict on on_progress=; checkpoint="rows.jsonl", which appends each row the moment it lands and, called again with the same path and tasks=, skips finished tasks and returns the union; and data.search["rollouts"] (landed, resumed, rerolled_by, lost_by, timed_out) with a UserWarning when re-rolls outnumber rows. With checkpoint= the --limit chunking above is optional: one call on all 601 prompts resumes from its own file.
  • Newer checkpoints, newer stack, and the weight sync is where it breaks. Qwen3.5-* (Qwen3_5ForConditionalGeneration) need vLLM >= 0.26 and transformers 5: --stack new in the trainer, --vllm 0.29.0 in serve_modal.py, both on a CUDA devel image because vLLM compiles kernels at start. transformers 5 names the text stack model.layers... and vLLM keeps it under language_model.model.layers...; TRL’s colocate weight sync passes names straight through, so before the rename in _guard_vllm_weight_sync every LoRA weight either crashed the run or was silently skipped (training a policy vLLM never saw). The guard prints any name it cannot place.
  • A 9B with linear attention needs three things to fit beside vLLM on one GPU. Qwen3.5-9B (gated delta net) went out of memory on an H100 and on an H200 before it fit: (1) flash-linear-attention in the image, or transformers falls back to reference kernels that took 138 GB for one micro-batch of 2 (causal-conv1d does not build in the image; its fallback is slow, not large); (2) a vLLM share that covers the weights plus a KV cache, --vllm-mem 0.40 for a 9B (0.25 leaves no room and the engine says “No available memory for the cache blocks”); (3) --micro-batch 1 --grad-ckpt. Gradient checkpointing is safe on --stack new because vLLM generates, not the HF path that broke on TRL 0.19. The cost: 12.7 min a step at 512 samples, so a 50-step round is about 11 hours on one H100.
  • Check what the SDK sent the model, not just what came back. Before 0.51, simulate(tasks=...) with a prompt-only agent drafted a tool surface for the situation writer and sent those schemas to the policy too. Qwen mostly ignored them; Nemotron-Nano-8B called a made-up tool on every task (pass@1 0.00), and 42 of r3’s 560 holdout replies were tool calls scored as failures. Pinned tasks now never draft tools; the r3 row below is the clean re-measure. The polluted files are kept in raw/with-drafted-tools/.

​
The holdout grew to 459 tasks (2026-09-17)

140 tasks gave a +-0.06 band, so a real 3-point gain could never be proven (whilehq/whileai-sdk#257). author.py wrote 1,482 more tasks the same way (rounds 2-11, every gold executed twice); the id hash puts 459 of the 2,223 in the holdout, about +-0.035 at k=4. The new tasks are harder (base hard 0.45 vs 0.56 on the old 140), so absolute numbers drop; the paired deltas are what to read. On the small holdout r4 vs base was +0.021 (-0.029..+0.068), “no change”. On 459 tasks the same two checkpoints give +0.026 with an interval that excludes zero: the gain was real and small, and the eval was too small to see it. Every later row is measured here. Round 5 is the batch experiment (whilehq/whileai-sdk#252): the same reward and prompts, but 512 samples per optimizer step instead of 8, only prompts the r4 policy solves sometimes (every non-unanimous group at k=8, 326 of 601, build.py --band 0.1,0.9), and length-truncated samples masked instead of scored 0. Twenty-five steps (12,800 samples, 3.5 h on one H100) moved the holdout as much as rounds 1-4 combined (8,000 steps, 32,000 samples). The step-25 row is served from the saved checkpoint through serve_modal.py on Modal rather than the hosted endpoint; the base was sampled through the same server as a control: base through that server is 0.53 (0.49..0.57), +0.004 (-0.019..+0.029) against the hosted base, so the serving path adds nothing. Steps 50, 75 and 100 follow. Step 50 is the result the lane was built to get. pass@1 0.53 -> 0.73 on 459 held-out tasks, paired, with the interval nowhere near zero, in 50 optimizer steps (25,600 samples, about 7 GPU-hours on one H100). Read the other columns before calling it capability: pass@4 barely moved (0.76 -> 0.79) while pass^4, all four samples right, went 0.25 -> 0.65. The policy did not learn many new queries; it learned to produce the query it could already find sometimes, every time, and to stop before the reply budget (replies with no query 13% -> 0%, SQL errors 12% -> 5%, median reply 1,400 -> 850 tokens). That is what a verifier reward on a 20-80% band does, and it is exactly the “reliable in production” property a customer is buying. Rounds 1-4 (one prompt per optimizer step, every prompt, truncation scored 0) spent 8,000 steps to gain 0.03; the batch, the band and the mask did 0.20 in 50. A second eval run of step 50 (another 1,836 samples) gives 0.73 (0.70..0.77), +0.207 (+0.181..+0.234) vs base: two of two. Steps 75 and 100 add nothing the interval can see (0.74, 0.74): the round saturated by step 50, which is 25,600 samples over 326 prompts, about four passes. The final adapter is while-ai/text-to-sql-shop-r5 on Hugging Face with the holdout rollouts of every checkpoint as eval-r5-step25/50/75 and eval-r5. Eval sets on the platform: step 25 ds_215b1d5320a2d04a, step 50 ds_99e38c55ac4eb56d, step 75 ds_b2fcbee3a747b4e3. What to run next, in order: the same recipe from the r5 adapter with the band re-measured on r5’s own rollouts (the prompts it now always solves drop out, the ones it never solves stay out); then the ceiling question, since pass@4 sits at 0.79 for every r5 checkpoint and pass@1 has caught up to it, more of the same reward cannot move this base much further. New capability needs a bigger base or harder training prompts it can solve sometimes.

​
Qwen3.5 on the 459-task holdout (2026-09-18)

Qwen3.5-4B and 9B are model_type qwen3_5 (Qwen3_5ForConditionalGeneration), which the 2025 pins cannot load: serve_modal.py --vllm 0.29.0 serves them and train_grpo_modal.py --stack new (transformers 5, TRL 1.13, vLLM 0.29) trains them; the lessons list says what that took. Same prompt, same 4,096 reply budget, same verifier, k=4. Every number in this table is scored on the first reply. The rows were first graded at 0.53 (9B) and 0.14 (4B) with “21% / 79% cut by the reply budget”: the engine had written a second user turn on 15% of the 9B rows and 65% of the 4B rows despite max_turns=1, because a reply holding a ”?” (plain-text reasoning does) earned a follow-up, written by the agent model itself, and the grader scored that second reply (whilehq/whileai-sdk#586, fixed). Cut each row at its first reply and re-grade, and the 9B base is 0.60, +0.053 (95% +0.022..+0.084) above Qwen3-4B after round 4, the strongest untrained start on this benchmark; the 4B is a usable 0.35. Neither emits <think> tags; both reason in plain text before the query. The second bug was in serving. Every checkpoint row read 0.61-0.62 against a 0.60 base because the LoRA served through vLLM was a no-op: PEFT under transformers 5 saves a Qwen3.5 (Qwen3_5ForConditionalGeneration) adapter as base_model.model.model.layers.N.*, vLLM keeps that text stack under language_model, and its LoRA loader activates nothing for a name it cannot place, without an error (whilehq/whileai-sdk#588; serve_modal.py now rewrites the header at copy time). The training log had said so all along: reward 0.90-0.98 on the band prompts while the “same” adapter scored 0.65 on them served (the base scores 0.66). Renamed, on the same 150 training prompts: base 0.60, as-saved 0.56, renamed 0.75. Served correctly, r1 step 75 is 0.82 (0.79..0.85) on the 459 tasks: +0.214 (95% +0.187..+0.242) paired against its base, easy +0.12, medium +0.26, hard +0.25 (0.53 -> 0.78), every interval above zero; pass^4 0.75, no reply cut off, 99.9% carry a query. Against the best Qwen3-4B (r5 final, 0.74): +0.081 (+0.056..+0.107). One 78-step round on a 9B did what five rounds on the 4B did, from a higher start. A second holdout pass on the same adapter scored 0.83 (0.79..0.86): +0.221 (+0.194..+0.249) vs the base, +0.088 (+0.063..+0.113) vs Qwen3-4B r5, and +0.007 (-0.003..+0.018) vs the first pass, so the verdict is moved, replicated x2. The step-25/50 and r2 step-25 rows, correctly served, are being re-run one at a time (a first attempt ran two eval jobs against one serve app and mixed adapters in one file; those rows were discarded). Round 2 stays stopped: its log showed 60-88% of groups at zero variance from step 1, the band cut on the base was solved by round 1, and the next round re-cuts it from the current checkpoint (202 of 601 prompts land in the 0.2-0.8 band on step 75, 118 shared with the old band). Adapters on Hugging Face: whileai/text-to-sql-shop-qwen3.5-9b-r1-step75 (PEFT layout, loads in transformers) and whileai/text-to-sql-shop-qwen3.5-9b-r1-step75-vllm (the vLLM layout served here). Holdout rollouts: eval-qwen3.5-9b-r1-step75-vllm in the dataset; the earlier eval-qwen3.5-9b-r1-step* configs measure the base and are kept for the record.

​
Other bases on the same holdout (140 tasks, k=4)

Served with serve_modal.py (vLLM on one L40S; --adapter volume:<run_id> serves a trained LoRA as <base>-adapter, --runs-volume names the volume the run was written to) and sampled through rollout.py --agent "vllm:<model>@<url>", so any Hugging Face model gets the same paired number as the hosted ones. Nemotron-Nano-8B-v1 is half of Qwen3-4B here, and its two arms are the same number because its reasoning mode never engages on these prompts: with the schema in the context (system turn, user turn, DDL only, question first, or the one-query rule softened to “think first”) every reply is a bare query, while the model card’s own math example thinks for 3,000+ characters. Even a forced <think> prefill closes after one line. Its failures are real SQL errors (WHERE NOT IN (...) with no column, an alias used before its join, non-grouped columns), not format. Headroom is 0.12, the same as Qwen’s. One GRPO round on it (text-to-sql-shop-nemotron-r1: 600 steps, 44 min on an H100, --steps-per-generation 8 --max-completion-length 512) is the first climb on this task whose interval excludes zero: +0.087 (95% +0.048..+0.130) paired over the 140 tasks, medium +0.10 (+0.01..+0.18) and hard +0.12 (+0.05..+0.20), easy flat. What it learned is mostly to write SQL that runs: executes 0.45 -> 0.65, SQL errors 0.55 -> 0.35, and the base was already at has_sql 1.00, so none of it is format. The SDK marks a single eval run per side moved_unreplicated, so each side was sampled three times (560 rows each): base 0.26 / 0.26 / 0.26, r1 0.35 / 0.36 / 0.37, paired deltas +0.087 (+0.048..+0.130), +0.098 (+0.055..+0.148), +0.111 (+0.068..+0.155). Three of three above zero. The adapter is served the same way as the base (serve_modal.py --adapter volume:run_f69e975a1571d445 --runs-volume zeroproof-train-runs, model id nvidia/Llama-3.1-Nemotron-Nano-8B-v1-adapter); eval sets ds_34f0fbd9337ab519 (base) and ds_2fbd036dd8597150 (r1) on the platform, Hugging Face configs eval-nemotron-8b-base / eval-nemotron-8b-r1 and adapter while-ai/text-to-sql-shop-nemotron-8b-r1. Read next to the Qwen table: the same reward, task set, and trainer moved a weaker base by nine points in one 44-minute round and a stronger base by three points in three rounds. The climb is real where the base leaves room below its own pass@4 and the failures are things a verifier can teach (SQL that does not run); it is slow where the base already writes valid SQL and the misses are semantics.

​
The hill climb (thinking on, GRPO, execution reward)

Measured through rollout.py --hosted <name> (the SDK path, agent_max_tokens=4096, timeout=300), 4 samples per task, paired by task. The holdout grew from 81 to 140 tasks when 324 more tasks were authored on the weak archetypes, so the intervals below are the 140-task ones (about +-0.06). r3 vs base: +0.029 (95% -0.016..+0.073), up at every difficulty (easy +0.02, medium +0.04, hard +0.03) and clearly up on one archetype, date and time (0.43 -> 0.58, +0.15, 95% +0.03..+0.27); the best checkpoint so far, not yet a proven climb by the SDK’s rule (the pass@1 interval still covers zero). This is the clean re-measure after the drafted-tools fix (see lessons); the first measurement, 0.62 (0.55..0.68) with 42 tool-call replies, is in raw/with-drafted-tools/. Each checkpoint’s holdout rollouts and adapter are on Hugging Face: dataset while-ai/text-to-sql-shop (configs eval-base, eval-r1, eval-r2, eval-sft-think, eval-r3, eval-r4), adapters while-ai/text-to-sql-shop-<checkpoint>. r4 vs r3: -0.007 (95% -0.048..+0.034); vs base +0.021 (-0.029..+0.068), hard +0.08 (+0.00..+0.17), easy -0.03. A second thousand steps of the same recipe kept r3’s gain and added nothing: the training reward sat at 0.65 through the whole round (it was 0.65 at the end of r3) and KL to the base stayed at 0.03, so the policy had stopped moving before r4 began. Round 4 is where “more steps” stops being the answer for this base; the levers left are the ones in the closing paragraph of this section (drop prompts the policy already always or never solves, 16 samples per prompt, a bigger base). The first two rounds did not move the holdout, while the training reward did climb (round 1 first-25-step mean 0.49 to last-25 0.63; round 2 up to 0.60-0.75 with KL 0.08), and thinking length fell from ~1,090 to ~800 tokens. That combination means the policy got better at the prompts it was shown and no better at held-out ones: 2,400 samples over 336 prompts, LoRA rank 16, is too small a dose for a 4B model to generalize SQL reasoning from, and the first thing GRPO learns is the cheap thing (shorter thinking, fewer failures to emit a query: has_sql 0.87 -> 0.90). What the numbers say to do next, in order: generate with vLLM inside the trainer (use_vllm, colocate) so a round costs minutes instead of 65 s a step, then run 5-10 epochs over the prompts with 16 samples each; only then judge the method. The table above is the product either way: every round is a paired number with an interval on the same holdout, so “it got better” is a claim the customer can check, and “it did not” is caught before anyone ships it.
Last modified on September 20, 2026