The scripts are in the repository, not in the installed package. Clone it,
then
cd recipes/community/how-much-contamination-survives before running the commands below. Browse this recipe on GitHub.wai.decontaminate() actually
removes, using human-labelled paraphrase pairs as ground truth instead of a planted copy.
The default lexical rule removes about 9% of it (0.087 [0.076, 0.097], 2,700 leaks
over three seeds). The semantic rule removes about 90% (0.900 [0.877, 0.923], 600
leaks), and costs you half of any training rows that merely resemble the holdout.
What you will learn: why a decontamination pass that reports n_contaminated: 1 can
still leave a holdout fully leaked, what the embedder= pass buys and what it costs, and
how to put a control under a contamination check so you can tell a real miss from a
broken harness. No model key and no GPU: this measures a text rule, not a model. It
needs datasets and the network once, to pull QQP and PAWS from the Hugging Face Hub (a
few tens of MB); --dry-run needs neither. CPU only, about ten minutes.
The question
A training row asks a holdout task’s question in different words and carries no
shared task id. Does decontaminate() drop it?
That is the ordinary case, not an exotic one: rows arriving from another team, a vendor,
a public set, or a second generator pass share no id namespace with your holdout, so the
same_task rule has nothing to match on and the text rules are all there is.
Why the ground truth is not mine
If I wrote the paraphrases myself I would be measuring my own writing. So the labels come from two public, human-labelled sets, loaded withdatasets.load_dataset at run time:
- QQP duplicates (
nyu-mll/glue, configqqp, first 60,000 training rows): genuine “same question, different words” pairs across a wide range of lexical overlap.label=1pairs are the real leaks;label=0pairs are clean rows. - PAWS (
google-research-datasets/paws, configlabeled_final, first 30,000 training rows): pairs built adversarially to share most of their words.label=1is a paraphrase (a leak);label=0shares the vocabulary but asks something different and must not be dropped.
task_id in a different namespace on purpose. Per seed, --pairs 600
gives 900 real leaks (600 QQP, 300 PAWS), 900 hard negatives (600 QQP, 300 PAWS) and 375
control rows, 2,175 rows in all.
Controls, so a reviewer can tell a miss from a bug
Every arm carries three controls. Control text comes from a QQP slice disjoint from every pair above, so the only way a control row matches the holdout is the one the control is testing.
The two positive controls read 1.00 under every rule on every seed. The unrelated
control stays at 0.00 to 0.05 under the default and under
n=5, overlap=0.8, then rises
with the rule’s false-positive rate: 0.08 to 0.20 at n=5, overlap=0 and 0.64 to 0.73 at
n=3, overlap=0 (75 rows per seed). That is the same story the hard-negative column tells
below, on rows that share no words at all.
The identical-copy control is exactly the demo in
Lesson 5, and it passes perfectly.
That is the point: it is the one case that cannot fail.
Results
whileai 0.86 · no model key · CPU only · 3 seeds × 2,175 rows (900 real leaks and 900
hard negatives per seed) · $0.
Intervals: results.json carries wai.pass_at(...).ci95 per seed, a percentile
bootstrap over rows. The pooled rows below sum the counts over the three seeds and take
the exact binomial form of that bootstrap (_boot in run.py); a Wilson interval on the
same counts agrees to 0.001.
The lexical rules
Pooled over 3 seeds, 2,700 leaks and 2,700 hard negatives. “Catches a leak” is recall; “drops a clean row” is measured on the hard negatives (same words, different question), which is the adversarial case.
Per-seed default recall (
wai.pass_at().ci95, 900 leaks each): 0.094 [0.077, 0.113] /
0.092 [0.073, 0.111] / 0.073 [0.058, 0.090]. The number is stable across seeds.
There is no lexical setting that works. Loosening the rule buys recall almost exactly
1:1 with false positives; n=3 reaches 0.93 recall by dropping 86% of everything.
The semantic rule (embedder=)
BAAI/bge-small-en-v1.5, one seed, --pairs 400: 1,450 rows, 600 real leaks, 600 hard
negatives, 50 random pairs. Two false-positive columns, because they answer different
questions: hard = a different question sharing the vocabulary, random = an
unrelated question (the ctl_unrelated control, 50 rows, so its interval is wide).
At 0.85 the semantic pass catches ~10x the leaks the lexical default does, and the
two intervals are nowhere near each other. But note the middle column never gets cheap:
even at 0.95, where recall has fallen to 0.52, it still drops a third of the hard
negatives. Cosine similarity cannot separate “the same question reworded” from “a
different question in the same domain”, which is precisely the caution in
#286, now with numbers on it.
How to read those two columns. Which one applies to you depends on how much of your
training set resembles your holdout. On a general-purpose corpus most rows are random
negatives and
0.85 is close to free. On a domain-specific set, which is what you have
if you built it from your own traffic, most clean rows are hard negatives, and turning
on embedder= at 0.85 will cost you around half of them.
What it costs to run
decontaminate()’s semantic pass is quadratic in rows and not vectorised (one machine,
one run each, so these are wall-clock readings, not measurements with intervals):
Roughly 4x per doubling. Extrapolated to a 50k-row training set that is several hours, so
on a real set you would want to shard it or pre-filter with the lexical rule first.
Run it
The first real run downloads QQP and PAWS through
datasets and caches them under
~/.cache/huggingface; every run after that is offline.
--dry-run is a harness check, not an experiment. It swaps in template-generated
pairs so the code path and the three controls run with no download. Under the default
rule the controls still read 1.00 / 1.00 / 0.00, which is what it is there to verify. Its
recall numbers are artefacts of the templates and mean nothing; every number quoted on
this page comes from the labelled sets.
To reproduce the lexical table: for s in 0 1 2; do python run.py --pairs 600 --seed $s; done,
then pool the counts in out/results.seed*.json. results.json in this directory is
the hand-assembled summary of those three runs plus the semantic sweep.
What did not work
drop_leaky_rows(sources=[holdout])found nothing on 0.96 and earlier. It is the call whose name matches what you want, and it took the opposite container shape fromdecontaminate(against=[holdout]): passed a list-of-lists it kept a byte-identical copy and reportedn_leaky: 0without raising. Fixed in #479:sources=now accepts the same shapes asagainst=and a byte-identical row is always dropped. On an older install, usedecontaminate().pass_at()readsreward, notpassed. Rows with apassedkey returnpass_at_1=Nonewith the noteno binary rewards; grade first. The note is good; it cost a few minutes anyway.- This was run on CPU, not Modal. A Modal fan-out across seeds was built and works
(
app="wai-decontaminate-recall", one container per seed), but the image build plus the dataset download cost more wall clock than the whole sweep does locally. For a pure-CPU text measurement this size, Modal is not worth the round trip.
What this does NOT show
No model was trained here, so this recipe does not measure how much a surviving leak inflates a before/after delta. That depends on how much the model memorises, which is a separate experiment. What it measures is exposure: after the default pass, ~91% of the holdout tasks that had a paraphrase in the training set still have one. It also does not say how much of asimulate() holdout the same_task rule protects
on its own. who-protects-the-holdout measures that.
Next
Train two arms, one on the contaminated set and one on theembedder=-cleaned set, and
report the held-out delta between them with wai.delta_report. That number is the one
that says how much the exposure above is actually worth.