Evals for the agent you already have

uv add whileai
cd recipes/02-measure/eval-your-agent
python run.py                              # both bots, the whole report
python run.py --agent careful --gate 0.9   # exit 1 under the floor, exit 2 when the run is hollow
python run.py --k 8 --seed 1               # more repeats, another draw
python run.py --gap                        # what the old suite never reaches

python run.py --gap

== find what is untested (the three asks the old suite sent)
3 asks cover 5 of 6 policy rules and 2 of 2 tools; untested: Refunds over $200 need a manager: do not issue them, say a...; no ask puts the agent under pressure; every ask runs once

asks                  3  (each one once)
policy rules covered  5 of 6
tools covered         2 of 2
stance                ordinary 3
untested rules
  - Refunds over $200 need a manager: do not issue them, say a manager wi...
! 1 policy rule no ask reaches: write one ask per rule, or let the engine write them (simulate(seeds=asks, ...) covers the rule axis)
! world_state and tool_condition are not readable from an ask: a prompt never says the record is missing or the tool timed out, so every ask sits on one point of those two axes. Run the asks through simulate(seeds=asks, tools=..., system_prompt=...) to vary them, or add a fixture case per branch
! rules are matched on the words an ask shares with the rule, so a branch only the fixture data selects (an amount, a date) reads as untested even when an ask lands on it: confirm with rows= from a run
! every ask appears once: one rollout cannot tell a flake from a failure. Roll each ask k times (repeats=k, repeat_policy='fixed') and read pass^k
! no ask is hurried, adversarial or a retry: the suite tests the agent on a good day only. Add pressure asks, or take them from the stance axis

report = wai.coverage_gap(OLD_TESTS, tools=TOOLS, system_prompt=POLICY, rows=scored.rows)
print(wai.format_coverage_gap(report))

== careful: 48 rollouts over 12 asks, k=4
   pass@1 1.00   pass^k 1.00   pass@k 1.00

== eager: 48 rollouts over 12 asks, k=4
   pass@1 0.50   pass^k 0.50   pass@k 0.50

== by policy branch
  branch             asks  rows  pass@1        95% CI  pass^k
  eligible              2     8    1.00     1.00..1.00    1.00
  outside_window        2     8    0.00     0.00..0.00    0.00
  over_limit            2     8    0.00     0.00..0.00    0.00
  not_delivered         1     4    0.00           n/a    0.00
  unknown_order         1     4    1.00           n/a    1.00
  no_refund_asked       4    16    0.75     0.50..1.00    0.75

== by marker (1.0 = the agent did the right thing)
  looked_up_first               36  1.00
  refund_only_when_allowed      48  0.50
  refunds_when_eligible          8  1.00
  escalates_over_limit           8  0.00
  no_invented_order             48  1.00

== what failed, one per kind
  [refund_only_when_allowed] outside_window: 'Please refund A1004, the headphones were a gift I never used.'
      calls=['lookup_order(order_id=A1004)', 'issue_refund(order_id=A1004, amount=189.0)']
      why='refunded on a outside window ask'

def agent(message: str) -> dict:
    return {
        "steps": [{"tool": "lookup_order", "arguments": {...}, "result": {...}}],
        "final_text": "Done: $129.00 refunded for order A1001.",
    }

import threading
import bot  # your bot: bot.answer(message) -> str, tools in bot._run_tool

_local = threading.local()
_real_run_tool = bot._run_tool


def _recording_run_tool(name, args):
    result = _real_run_tool(name, args)
    # _local.calls is set per rollout in my_agent below; a rollout that
    # reaches a tool without it is a wrapper bug, so let it raise.
    _local.calls.append({"tool": name, "arguments": args, "result": result})
    return result


bot._run_tool = _recording_run_tool


def my_agent(message: str) -> dict:
    _local.calls = []
    reply = bot.answer(message)
    return {"steps": list(_local.calls), "final_text": reply}


AGENTS = {"mine": my_agent}