whileai.simulations.monitor

Watch a live training run for reward hacking. 2 public names. import whileai.simulations as wai, then wai.name.

Name	What it does
`HackMonitor`	Watch a TRL run for reward hacking.
`format_hack_monitor`	The block a person reads: the curve, then the alarms.

HackMonitor

HackMonitor(
    run: TrainingRun | None = None,
    holdout: Sequence[Any],
    proxy: Callable[..., Sequence[float]] | None = None,
    gold: Callable[[dict], Any] | None = None,
    sample: Callable[..., list[list[str]]] | None = None,
    every: int = 10,
    k: int = 4,
    n_prompts: int = 32,
    window: int = 3,
    delta: float = 0.1,
    length_pct: float = 0.25,
    kl_budget: float | None = None,
    endorsed: Sequence[str] = (),
    stop_on: str | Sequence[str] = (),
    buffer: int = 512,
    max_new_tokens: int = 256,
    concurrency: int = 8,
    seed: int = 0,
    n_boot: int = 500,
    n_perm: int = 50,
    scan_min: int = 8,
    sampling: Mapping[str, Any] | None = None,
)

Watch a TRL run for reward hacking. See the module docstring. holdout is a list of prompts (strings or message lists) or rows ({"prompt": ..., <extra columns>}); extra columns reach the proxy as keyword lists, the way TRL passes dataset columns. proxy is a TRL-shaped reward function (f(prompts=, completions=, **cols)); leave it None to use the reward function passed through wrap. gold is a judge under the SDK contract (a rollout row in, a reward or {"reward": ...} out). sample overrides how completions are drawn: sample(model, tokenizer, prompts, n=, max_new_tokens=) -> list[list[str]]; the default uses the chat template and model.generate; sampling (temperature, top_p, batch) steers that default sampler. n_boot is the bootstrap count behind the gold-vs-window interval, n_perm the permutation count behind the feature scan, scan_min the fewest buffered completions the scan runs on. Every number has its reason in defaults.py (MONITOR_*). run is the platform run the points and alarms land on; None keeps everything on the monitor (history, alarms, summary()).

HackMonitor.evaluate

evaluate(
    self,
    step: int,
    model: Any = None,
    tokenizer: Any = None,
) -> dict[str, Any]

Sample the holdout from the live policy and score it both ways. Appends to history, logs to the run, checks the alarms. Returns the history entry.

HackMonitor.summary

summary(self) -> dict[str, Any]

What the run page and finish carry: the curve points, the alarms, the last scan, where it stopped.

HackMonitor.wrap

wrap(
    self,
    reward_fn: Callable[..., Sequence[float]],
) -> Callable[..., Sequence[float]]

The reward function, watched: every completion it scores goes into the buffer with its reward. Pass the result to the trainer as reward_funcs. The name survives, so TRL’s rewards/<name> column does too.

format_hack_monitor

format_hack_monitor(summary: Mapping[str, Any]) -> str

The block a person reads: the curve, then the alarms.

Get started

Guides

Concepts

Reference

API

whileai.simulations.monitor

HackMonitor

HackMonitor.evaluate

HackMonitor.summary

HackMonitor.wrap

format_hack_monitor

​HackMonitor

​HackMonitor.evaluate

​HackMonitor.summary

​HackMonitor.wrap

​format_hack_monitor

HackMonitor

HackMonitor.evaluate

HackMonitor.summary

HackMonitor.wrap

format_hack_monitor