The scripts are in the repository, not in the installed package. Clone it,
then
cd recipes/02-measure/pass-at-k before running the commands below. Browse this recipe on GitHub.
The SDK computes all three from the per-ask pass-rate vector, using the
unbiased estimators (Chen et al. 2021 for pass@k, tau-bench for pass^k).
Below
repeats=4 the k-way numbers are withheld with a note rather than a
figure too noisy to act on.
Run it
ci95 on
the PassAt). Twelve asks is a small sample and the interval says so;
report it next to the point estimate.
The seed decides the draw: the run passes reproducible=True, so the same
--seed prints the same numbers at any --concurrency. Change the seed and
the numbers move; the shape is the point. This agent is mostly consistent
(right or wrong every time on eleven of twelve asks), so the honest verdict
is that repeats alone will not teach it much: the one mixed ask is the only
gradient, and the six it never passes need harder cells or demonstrations,
not more rollouts.
Reading it
The histogram is the part the mean hides. An agent at pass@1 = 0.5 could be right on half the asks every time (no RL headroom, a coverage problem) or right half the time on every ask (all headroom, a consistency problem). pass^k and pass@k tell those apart; the mean does not.- pass@1 is the headline. The agent runs once in production. Report it, track it across prompt changes and model upgrades, and let the other two explain it.
- pass^k is the contract. Use it as the number you promise. If pass^k sits far below pass@1, the agent knows how but does not do it reliably; that is a consistency target, and mixed groups are the training data for it.
- headroom sizes the RL run.
recommend(mode="rl")turns the mixed rate into a rollout budget;select_for_rlkeeps the mixed groups whole. Headroom near zero means nothing to learn from these asks: harder cells or a stricter judge before more rollouts.
Where else it shows up
data.pass_at on any simulation, .pass_at on the ScoredData from
data.grade(judge=...) and evaluate(...), the same keys in
group_signal(rows), and pass_at in the .meta.json sidecar of
save(meta=True).
One caveat for LLM judges: pass@k inflates on judge false positives and
pass^k on false negatives. pass@1 is the least sensitive of the three, which
is the other reason it stays the headline.