> ## Documentation Index
> Fetch the complete documentation index at: https://docs.withwhile.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Install with `uv add whileai`; import as `import whileai as wai`.
> Run the offline path first (`simulator=False`, `wai.seeded_agent`, a callable judge); no key is needed for it.
> Report every pass rate with its interval and n, as `scored.pass_at` prints it.

# Rows: the contract every measurement reads

> The five row keys pass_at, compare, eval_variance, holdout_size, decontaminate and select read, which are required, and wai.rows(), the one call that builds them from a public benchmark.

A row is one completion of one prompt with the verdict on it. `simulate()`
emits rows; so does `wai.rows()`, from prompts and completions you already
have (a public benchmark, a model's answers to it). Every measurement call
reads the same five keys, so the rows from either source go into every call
unchanged.

## The five keys

| key          | required                             | what it is                                                                                                                                                                                                                |
| ------------ | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `task_id`    | no, defaults to a hash of the prompt | what the completions of one prompt group under. Every interval is over tasks, never rows (Miller 2024, arXiv:2411.00640), so ten rows with one `task_id` are one task. The engine writes the same value as `scenario_id`. |
| `prompt`     | yes                                  | the prompt text. `decontaminate` matches on it by default.                                                                                                                                                                |
| `final_text` | yes                                  | one completion. A verifier reads it as the candidate.                                                                                                                                                                     |
| `reward`     | for everything but `decontaminate`   | a number in \[0, 1]. 0 and 1 are the binary outcome `pass_at` counts and `select` bands on; anything between is a partial score those two skip.                                                                           |
| `markers`    | no                                   | `{name: number}` per completion. **Input as well as output.** `compare(proxy="marker:name")`, `eval_variance` and `judge_trust` read them wherever they came from: a judge wrote them, or you did.                        |

A row may carry more (`messages`, `steps`, `judge_name`, `lineage`,
`privileged`) and the calls use what they find. Nothing else is required.
The typed view is `whileai.simulations.schema` (`Task`, `Rollout`,
`Judgment`, `Marker`); `wai.rows()` builds through it, so its rows carry
`schema_version` and the same shape a run writes.

## Build them: `wai.rows()`

```python theme={"theme":"vitesse-dark"}
import whileai as wai

questions = ["What is 2 + 3?", "What is 7 * 6?", "What is 10 - 4?", "What is 9 / 3?"]
gold = ["5", "42", "6", "3"]
answers = [["5", "4", "5", "5"], ["41", "41", "42", "40"], ["6"] * 4, ["3", "2", "3", "3"]]

rows = wai.rows(questions, answers, wai.verify.MathEqual(), references=gold)
print(rows[0]["task_id"], rows[0]["reward"], rows[0]["judge_name"])
```

* `prompts`: strings, or message lists (`[{"role": "user", "content": ...}]`);
  the row's `prompt` is the last user turn and `messages` keeps the list.
* `completions`: one string per prompt, or one sequence per prompt. The
  sequence is k completions of the same prompt, which is what `pass_at`'s
  k-way numbers and `select(mode="rl")` need.
* `reward`: a verifier (`wai.verify.MathEqual()`, `Numeric`, `ExactMatch`,
  `CodeExec`), a callable `(prompt, completion)` or
  `(prompt, completion, reference)` returning a number in \[0, 1], a
  judge-contract callable `(row) -> verdict`, or the numbers themselves,
  nested like `completions` or flat. A verifier or callable runs through
  the same path `data.grade()` uses, so the row says what scored it.
* `references=`: the gold per prompt. It lives under `privileged.reference`,
  where a verifier reads it and no training export projects it.
* `task_ids=`, `markers=`: names per prompt, measurements per completion.

Precomputed scores and your own markers:

```python theme={"theme":"vitesse-dark"}
scores = [[1, 0, 1, 1], [0, 0, 1, 0], [1, 1, 1, 1], [1, 0, 1, 1]]
lengths = [[{"chars": len(a)} for a in group] for group in answers]
given = wai.rows(questions, answers, scores, markers=lengths)
print(given[1]["reward"], given[1]["markers"])
```

## Then measure

Everything downstream takes the list as it is.

```python theme={"theme":"vitesse-dark"}
tuned = wai.rows(
    questions,
    [["5"] * 4, ["42", "42", "42", "41"], ["6"] * 4, ["3"] * 4],
    wai.verify.MathEqual(),
    references=gold,
)
print(wai.pass_at(rows))  # pass@1 with its interval, pass^k, pass@k
print(wai.select(rows, mode="rl", band=(0.2, 0.8)))  # the groups that carry gradient
print(wai.compare(rows, tuned))  # is the change real
```

`select(mode="rl")` keeps whole groups the model passes between 20% and
80% of the time (Lambert 2025, chapter Reasoning; DAPO, arXiv:2503.14476)
and drops the rest: a group all-pass or all-fail has no advantage to learn
from. Its report names every gate that dropped a row, so a benchmark whose
rows all vanish says why.

The worked example, 200 GSM8K test questions through `MathEqual`, three
eval passes, `holdout_size` and a `compare` report, offline, is
[`recipes/02-measure/public-benchmark`](/recipes/02-measure/public-benchmark).

## What the calls read

| call                    | groups by                     | reads                                 | skips                                                          |
| ----------------------- | ----------------------------- | ------------------------------------- | -------------------------------------------------------------- |
| `pass_at`               | `task_id`                     | binary `reward`                       | partial and unjudged rows                                      |
| `compare`               | `task_id`, paired across arms | `reward`, every shared `markers` name | tasks present on one side                                      |
| `eval_variance`         | `task_id` per run             | `reward`, `markers`                   |                                                                |
| `holdout_size(before=)` | `task_id`                     | `reward`                              |                                                                |
| `decontaminate`         |                               | `prompt` (`fields=`)                  |                                                                |
| `select(mode="rl")`     | `task_id`                     | binary `reward`, `final_text`         | non-binary rewards, empty or cut-off replies, unanimous groups |