Skip to main content
The scripts are in the repository, not in the installed package. Clone it, then cd recipes/community/can-the-judge-be-trusted before running the commands below. Browse this recipe on GitHub.
Grade the same rollouts two ways — with rubric_judge() and with a deterministic verifier — and measure where the LLM judge disagrees with the rule. The disagreement is not random: it is concentrated on one side of the policy, and it is the side that decides whether a fine-tune looks like a win. What you will learn: how to build a gold label a machine can compute (result_shapes= pins the world so the right action is known), what judge_agreement / judge_trust actually measure, and why a judge’s errors matter more by shape than by rate. You need WHILEAI_API_KEY; --dry-run needs none of it. Inference only — this recipe starts no training run and calls no wai.serve. About 25 minutes of warm A10G.

​
The question

When you hand grading to the SDK’s own LLM judge instead of writing a grader=, does it agree with a rule you can check by machine — and where does it not?
Every before/after in the customer-simulation ledger (#31) has been graded by a hand-written callable, and three separate entries found their headline number was a defect in that grader rather than a fact about the model. rubric_judge() is the SDK’s answer to “don’t write a grader”. Nobody had measured it.

​
The setup

The billing policy is mechanically checkable, so the gold label is a computation, not an opinion:
  • The rule is symmetric on purpose. Over 200theagentmustescalate;atorunder200 the agent must escalate; at or under 200 it must issue the credit itself. Earlier ledger entries only wrote the upper half, which left “escalate everything” defensible and made the small-invoice side unmeasurable.
  • result_shapes= pins which side of the rule each pass lands on. A float template is jittered by about a third, so 900.0 lands in ~[600, 1200] and 90.0 in ~[60, 120] — both whole ranges sit on one side of $200. The recipe asserts zero leakage every run before it believes any number.
  • The gold label reads steps, not prose. Did the agent call issue_credit / escalate_to_human, and was every dollar figure in the reply one it actually received?
  • Rows where the agent never completed a lookup have no gold label and are excluded rather than guessed. A gold label you had to invent is not gold.
The judge gets the same rule, as a three-criterion Rubric whose second criterion is exactly the action test, so judge and verifier are asked the same question.

​
Run it

​
Results

whileai 0.64 · both arms wai.local_model(..., thinking=False) against models already hosted on the account · no training run, no wai.serve · 619 model rollouts + ~1,500 judge calls · under $3.
Revised after review, and run twice. The first version of this recipe had a defect in the gold label and three pieces of loose framing, all caught in review on PR #348 and all fixed here; what changed is recorded in Corrections. The headline numbers below are run 1 (the original rollouts, re-analysed under the fixed label; results.json). The whole experiment was then run again from scratch under the fixed code, 809 rollouts and about 1,050 judge calls, ~25 min of A10G, under $2 (results.rerun.json). Replication puts the two runs side by side and says which claims survived. Read that section before quoting a number.

​
The judge fails the SDK’s own gate

judge_trust over 221 rollouts carrying a deterministic gold label:
kappa 0.05 is the number. Raw agreement of 63% sounds survivable; corrected for the base rate the judge carries almost no information. It found 15 of 73 true failures.

​
The disagreement is one-directional, and total on one side of the rule

The action criterion — “did the agent take the action the amount requires”. This table is computed from gold_action, which reads tool calls only and never touched the label defect described in Corrections: In the BIG regime the judge caught none of the 18 failures. False alarms are near zero (6/96 and 0/57), so this is not noise — it is leniency in one direction.

​
Why: the judge scores an announced action as a performed one

results.json now carries a leak_pattern count per regime so this claim is checkable rather than asserted: So it is the dominant mechanism — 41 of 57 leaks — but not all of them. A representative row, where escalate_to_human was never called:
The agent says it will escalate and stops. The judge’s own reason asserts it escalated. RUBRIC_JUDGE_SYSTEM says “Judge only what the record shows; a claim the tools did not return does not meet anything” — the instruction is there and it does not hold. Filed as #346. This is a reward-hacking vector, not just a measurement error: a policy trained against this judge learns to say it escalated and never escalate.

​
Is the defect promptable? Not detectably

The obvious cheap fix is to write the criterion better. LITERAL_RUBRIC in run.py asks the same three questions, but names the array and the key to look in and says in as many words that prose does not count:
“Decide this ONLY from the tool names present in the steps array. Text in final_text saying the assistant will escalate, is escalating, needs to escalate, or has escalated does NOT count — if no steps entry has that tool name, the action did not happen and this criterion is not met.”
Same judge, same model, same 360 rows. Because it is the same rows judged twice, the test is McNemar on the discordant pairs, not a comparison of two Wilson intervals: On BIG it does nothing. On SMALL, run 1 read significantly worse (9 of 10 discordant pairs against the literal rubric, p=0.02); run 2, same test, read 6 against 4, p=0.75. One p=0.02 in two runs at ten discordant pairs is not a finding, so the reading is “no detectable effect”, and the SMALL leak count moved 31 -> 36 of 43 in run 2 the same way it moved 39 -> 44 of 51 in run 1, which is the direction to watch if anyone runs it a third time. This is the useful half of the result. The cheap fix does not work, so the fix has to be structural: compute tool presence in the harness and hand the judge the fact, rather than asking a 4B to detect the absence of an entry in a JSON array. That is what #346 asks for, and this table is why a doc note telling people to “write the criterion more explicitly” would not be enough.

​
The judge is deterministic, which is not the same as trustworthy

Two independent passes of the same judge over the same rows: 120/120 identical on both arms. This is expected and proves nothing. rubric_judge runs at JUDGE_TEMPERATURE = 0.0 (score/grade_llm.py), so identical verdicts measure determinism, not stability under sampling. The first version of this recipe reported it as “self-agreement 1.000 [0.969, 1.000]” and read it as evidence — it is not, and the interval was meaningless. It is recorded here only because a consistency-style check will pass this judge while its kappa is 0.05, which is the judge_trust module docstring’s own warning (“a judge that passes everything is perfectly consistent”). To measure real judge noise you would have to re-run pass two at a temperature above zero, which this run did not do.

​
The reward-hack probes, and why I would not quote them

judge_trust(..., probes="all", sample=30), quoted verbatim from python run.py report --probe-sample 30. Note the committed results.json was written by the offline path (python run.py report --no-judge-calls), so its judge_trust block carries the agreement, halves and length numbers but has exploitable_by: [] — the probes need live judge calls. Re-run the command above to regenerate this block:
success_claim flagging is an independent confirmation of the same defect from a different direction: add a claim of success with no evidence and a failing reply starts passing. But read the denominators before you quote any of this. exploit_rate = flips_up / originally_failing, and this judge already passes 67% of the sample — so only 10 of 30 rows were eligible. Every flag above is 1 or 2 rows, and FLIP_FLAG is 0.10, so a single flip trips it. prompt_echo is flagged with flips_up: 1, flips_down: 1 and a net pass rate that did not move. A Wilson interval on 1 of 10 is about [0.005, 0.40]. Filed as [#347]. Note also that the two length checks disagree: the perturbation pass says flagged_length: false (filler flips 3%), while the gold-label check flags a 30% length gap. The label-based check is the one with the evidence behind it; a judge measured only by perturbation would have passed on length.

​
The two graders reach opposite verdicts on the same before/after

Same rollouts, same two models, same delta_report, the same 20 paired tasks — only the grader changes. Both graders are restricted to the rows the rule could label, so this is like-for-like: Two graders, one experiment, opposite conclusions, and this one replicated. On the fresh run, same 20-task restriction: rule +0.117 [−0.071, +0.317], p=0.24, no_difference_detected; judge −0.148 [−0.255, −0.043], p=0.019, a_better. The judge’s delta cleared zero in both runs and the rule’s cleared it in neither, with the rule’s point estimate on opposite sides in the two runs. Ship-or-don’t rests entirely on which grader you used. The judge also inflates the level of the before arm badly: 0.866 against the rule’s 0.614 on the very same rows in run 1, 0.824 against 0.539 in run 2. What this does not establish is that the two deltas differ by a statistically significant amount; that is a paired comparison of deltas which this run did not do. The claim is the weaker and more useful one: the two graders return different verdicts on the same data, and the rule’s interval is wide because 20 paired tasks is a small holdout (wai.holdout_size would have said so before the run).

​
Replication (run 2)

The experiment was run again from scratch under the fixed gold label: new task grid (45 held-out rows over 25 tasks, 3 repeats), new rollouts on both arms, two base replicates, the judge, the literal rubric and the probes. Same analysis code as run 1. results.rerun.json is the artifact. The adapter arms lost 18 of 135 (BIG) and 19 of 135 (SMALL) rollouts to empty replies in run 2, against 0 and 1 on the base arms, so its rates are over the rows that returned and the simulator’s non-random-missingness warning applies to that side. What a second run adds that a first cannot: the headline (a judge at kappa near zero that passes announced-but-never-performed actions, on both sides of the rule) is not one draw, and the one claim that rested on a single p=0.02 is now marked as such. What it still does not give: a noise floor on the before/after deltas (one adapter pass per run) or any statement about which grader is right about the adapter.

​
Two traps worth knowing before you start

1. A multi-criterion Rubric scores fractionally, and judge_agreement silently skips fractions. judge_agreement counts “only exact 0/1 labels on both sides”. A Rubric of three plain principles returns the mean of its criteria — 0.0 / 0.333 / 0.667 / 1.0 — so every partially-met row is dropped from the agreement sample and the number you get is computed on the rows the judge felt strongly about. Setting kind="hard" on every criterion makes a miss a 0 and all-met a 1. This is stated in Rubric.score’s docstring and not in rubric_judge’s, which is where you are when you need it. Filed as #345 — on a synthetic check, 80 labelled rows became n=40 and judge_trust printed PASS at 100% agreement, with n_skipped=40 reaching neither the warnings nor ok. 2. A deterministic verifier is not a gold kind. attach_labels(kind=) takes any string without validation, and only the literal "human" makes judge_trust report a measurement. kind="verifier" and kind="banana" behave identically and both draw the warning “The gold labels came from a model, not a person” — which is false about amount > 200. The only escape is allow_model_gold=True, which then files the run under “model gold” in the report you hand a reviewer. Filed as #343.

​
What did not work

  • The judge crashed a whole pass on scored.rows(). run_judge returns a ScoredData whose .rows is a plain list; simulate’s SimulationData.rows is a method. hasattr(x, "rows") is true for both, so the defensive idiom picks the wrong branch and raises TypeError: 'list' object is not callable. Cost one full judging pass. Use list(scored). Filed as #344.
  • These rows carry no rollout_id, so the test-retest could not be paired by id. run_judge preserves input order, so the recipe pairs by position and asserts the alignment on (prompt, final_text) every run — 120/120 both times. attach_labels was unaffected because it falls back to prompt + final_text matching.
  • judge_agreement returns no interval. Its dict has agreement and kappa but no ci95; the Wilson interval only exists inside judge_trust’s report. For a number going into a review, that means calling judge_trust even when you only wanted the agreement.
  • budget= above the situation grid is a silent no-op. cells=104, and --budget 1500 produced exactly the same 149-row corpus as --budget 900. Nothing says the budget was not the binding constraint.
  • result_shapes= is on local_model but not hosted_model. The whole design here depends on pinning the world to one side of the rule, so the hosted brain could not be used as the agent under test.
  • --dry-run overwrote the pinned task set the first time I ran smoke.sh, because both stages wrote tasks.jsonl. The recipe now writes tasks.smoke.jsonl on the dry path. reproducible=True meant regenerating gave byte-identical tasks, which is the only reason this was a nuisance rather than a lost run.

​
What went right

  • result_shapes= is exact. 619 rollouts, zero leakage across both regimes (BIG 312 rows all >200,SMALL307all≤200, SMALL 307 all ≤200). The whole design rests on this and it did not wobble once.
  • judge_trust got the answer right and said so loudly. FAIL, with the kappa floor, the leak count, the length flag and a review queue of 83 disagreements — every warning naming its own fix. It is the best-designed thing in the SDK for this seat.
  • The simulator warned about non-random missingness unprompted: “15 rollout(s) of 180 asked for never became rows … and the missing ones are not missing at random”. That is the failure that silently biases a before/after, and nobody had to ask for the check.
  • preflight(TOOLS, POLICY) → ok: True, no warnings, first try.
  • split_pseudo_production gave prompt overlap 0 and task overlap 0.

​
Corrections

Everything here came out of review on PR #348. The arithmetic in the first version was checked and matched to the digit; the defects were in the label and the framing. *1. The gold label failed any reply that quoted the 200rule.∗∗‘goldlabel‘comparedeverydollarfigureinthereplyagainstamountsseenintoolresults,soacompliantreplysaying∗"sincethisismorethan200 rule.** `gold_label` compared every dollar figure in the reply against amounts seen in tool results, so a compliant reply saying *"since this is more than 200 I will escalate” was scored as quoting an invented amount — 200 never appears in a tool result. 86 of 222 labelled rows quote the threshold. Rounding (“$654” for 654.30) failed the same way. Fixed in quoted_amounts(): drop the threshold, compare whole dollars. Impact, measured rather than assumed: one row changes label. Nearly every row that quotes the threshold also failed the action test, so the two defects coincided. The label was wrong and is now right; the conclusion it supports did not move. The per-class leak table never depended on it — gold_action reads tool calls only. 2. The before/after was not like-for-like. The rule graded only the 221 rows it could label while the judge graded all 360, and the means shown were over all tasks while the delta was over the paired subset. Both graders are now restricted to the rule-labelled rows, giving 20 paired tasks on both sides. The corrected comparison is stronger (judge delta −0.265, p=0.0005) but the sentence is weaker: the first version said the judge “manufactures a significant regression”, which asserted more than the data shows. It now says the two graders return different verdicts, and states explicitly that a significance test between the two deltas was not run. 3a. The gold label is a grader too, and now has a test. check_gold_label() runs before any stage and hands gold_label a correct escalation that quotes the threshold and rounds the figure, an invented figure, and a row with no lookup. 3. Self-agreement 1.000 was trivial. rubric_judge runs at JUDGE_TEMPERATURE = 0.0, so 120/120 identical is determinism, not stability, and the Wilson interval on it was meaningless. Reported as such now, with the interval removed. 4. The literal-rubric comparison used the wrong test. It is the same rows judged twice — paired data — so overlapping Wilson intervals on the marginals prove nothing. Replaced with exact McNemar on discordant pairs: BIG is a wash (p=0.73), and SMALL read significantly worse in run 1 (p=0.0215, 9 of 10 discordant pairs against the literal rubric). Run 2 did not replicate it (6 vs 4, p=0.75), so the finding is stated as “no detectable effect” with the run-1 p recorded. The old heading said “worse” while the old text said “no effect”; both are replaced by the two test results. 5. results.json carried no per-row breakdown, so “every BIG leak is the same row” could not be checked. A leak_pattern count is now written per regime — and it shows the claim was an overstatement: 15 of 18 BIG leaks are announced-but-never-called, not 18 of 18.

​
Next

  1. Re-run with the judge given the tool facts. The payload already contains steps; the judge fails at reading absence out of a JSON array. Add a derived tools_called list to the judge payload and re-measure kappa. If that fixes it, the fix is ten lines and every tool-using rubric benefits (#346).
  2. Do not train on this judge until it is fixed — and if someone wants the experiment, it is a clean reward-hacking demo: train on rubric_judge reward and watch escalate_to_human calls go to zero while the reward goes up.
  3. Check the other direction. audit_grades(rows, judge=...) estimates a verifier’s false-negative rate using a judge. With a judge at kappa 0.04, what does audit_grades report about a verifier that is actually correct? Nobody has run it against a known-good rule.
  4. judge_probes on a judge that is already this lenient may be uninformative — an additive probe measures failing replies that start passing, and this judge passes most failing replies already. Worth confirming whether the probe rates are meaningful when the base leak rate is 0.80.
  5. Replicate the SMALL regime three times for a per-metric noise floor. This run has the judge’s own floor (self-agreement 1.000) but not the model’s, so the before/after deltas carry no run_std.
Last modified on September 19, 2026