Skip to main content
How simulate() makes evals and training data. Combinatorial coverage of situations, a sandbox world with failure modes, and a judge validated before training. The longer read is Simulations; the short version with references is the engine under Concepts.

​
Eight steps

​
What is ours

  • Covering array. Every pair of axis values appears together at least once. Most failures are two-factor interactions [6].
  • Arm search. structured, llm_guided, open_ended, behavior_targeted, failure_mutation, weighted each batch by the yield of new behavior signatures. Novelty search, not importance sampling [7].
  • Sandbox world. Built from the tool JSON schemas. Deterministic per seed. Unknown id: not found. Schema-echo argument: refused. Every number it answers with is a named default (defaults.py) and a WorldOptions field.
  • Cuts. SFT: reward=1 rows, loss mask on agent turns. DPO: pairs with margin and length gap. GRPO: mixed groups, 20 to 80 percent band [4, 5]. RM: all graded rows.

​
What we take from the literature

  • pass@1 / pass^k / pass@k. Headline, reliability, RL headroom. Unbiased combinatorial estimators over k samples per task [2, 3]. score/passat.py.
  • Intervals. Bootstrap over tasks, not rollouts. Before and after is a paired difference with a sign-flip permutation p [1]. Ship when the interval excludes zero. score/stats.py.
  • Judge. Agreement and kappa against gold labels, Wilson interval, held-out halves, length perturbation, probes. Different model family than the policy [8]. Rubric hash on every label. score/judge_trust.py.
  • Hack scan. Var(r) = E[Var(r | task)] + Var(E[r | task]). Only the first term is GRPO gradient. The top within-task feature is compared to a permutation floor from reward shuffled within task [9]. score/hack_scan.py.

​
Questions

Do you use importance sampling? No. Importance sampling corrects an estimator for a wrong proposal distribution; we are not estimating production, we are covering the failure space. Each row keeps its logprobs, policy version and sampling settings, so an asynchronous trainer can form the truncated ratio exp(log pi_new - log pi_old) itself [10, 11]. score/logprobs.py and score/reference.py score the same tokens under a reference model for the KL side. SFT or RL? Both, from the same graded rows. select_for_sft keeps each prompt’s best reward=1 row and spreads its picks across behavior signatures, with a loss mask on agent turns. build_preference_pairs takes length-matched pairs with a margin. select_for_rl takes whole groups. A reward model takes all graded rows. Isn’t the judge just another LLM? Yes. So it is measured against gold labels (judge_agreement), probed with known hacks (judge_trust), versioned by rubric hash, and drawn from a different model family than the policy [8]. How do you know training helped? delta_report: paired before and after on held-out tasks with a bootstrap interval. Markers such as argument_grounding catch regressions pass@1 hides; a must_not_regress marker whose interval sits below zero fails the run [1].

​
References

  1. Lambert, N. Reinforcement Learning from Human Feedback. 2025. rlhfbook.com.
  2. Chen, M. et al. Evaluating Large Language Models Trained on Code. arXiv:2107.03374, 2021.
  3. Yao, S. et al. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045, 2024.
  4. Shao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024.
  5. Yu, Q. et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476, 2025.
  6. Kuhn, D. R., Wallace, D. R., Gallo, A. M. Software Fault Interactions and Implications for Software Testing. IEEE Transactions on Software Engineering 30(6), 2004.
  7. Lehman, J., Stanley, K. O. Abandoning Objectives: Evolution Through the Search for Novelty Alone. Evolutionary Computation 19(2), 2011.
  8. Zheng, L. et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS, 2023.
  9. Gao, L., Schulman, J., Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML, 2023.
  10. Schulman, J. et al. Proximal Policy Optimization Algorithms. arXiv:1707.06347, 2017.
  11. Noukhovitch, M. et al. Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models. ICLR, 2025.
Code paths are relative to whileai/simulations/.