A benchmark score means "this model can do the task" only if the model hasn't seen the task. Once it has, the same number means something else: "this model remembers the task." Those are different capabilities, and you can't tell them apart from the score.
That's the whole contamination problem in two sentences. Everything else is plumbing.
Two leaks, not one #
There are two ways an answer gets into a model during an eval.
The first is live access. The model is given tools — search, browsing, a code sandbox with network — and during the eval it finds the benchmark, a solution write-up, or a discussion thread. This leak is new with agentic evals, and it's the one that has prompted labs to cut live internet from evaluation runs.
The second is training exposure. The benchmark questions, or their answers, or close paraphrases, were in the pretraining data. The model doesn't need to search; it already knows. This leak is older, quieter, and much harder to close.
Cutting the network fixes the first. It does nothing for the second. And a lab that announces it has "removed internet access from evals" has told you about the leak that was easier to plug.
Why live access was never a test #
I'll be blunt about the first leak, because I've argued it on X: a capability eval that can reach the open internet isn't a test. It's a production deployment with a scoring rubric attached. If the model can look up the answer, you're measuring search plus synthesis, which might be a fine thing to measure, but it isn't what the benchmark's name claims.
There's also a safety angle. An eval for a dangerous capability, run with network access, is the model actually exercising that capability against the real world. Calling it a test doesn't change what the packets did.
So, yes, cut the network. Then don't stop there.
The training leak is the bigger one #
Pretraining corpora are scraped from the public web. Public benchmarks live on the public web. Unless someone actively filters them out — and filtering is imperfect, because paraphrases, translations, and solution blogs don't match on exact strings — some benchmark content ends up in training.
What makes this hard is that the effect is invisible from inside the eval. A contaminated model doesn't behave differently in an obvious way. It just does a bit better on the items it has seen. Aggregate that over a benchmark and you get a score that looks like progress.
How to tell the difference #
You can't inspect a model's memory directly, but you can design evals that make memory useless:
- Held-out sets that were never public. Keep a private split. If the public score and the private score diverge, that gap is your contamination estimate.
- Fresh items, dated after the training cutoff. Questions written after the model's data was frozen can't have been memorized. This is the cleanest control there is.
- Perturbation. Change names, numbers, and surface details while keeping the underlying problem identical. A model that reasons should be roughly indifferent. A model that memorized should drop.
- Canary strings. Embed unique markers in benchmark files so their presence in training data can be detected later.
None of these is perfect alone. Together they turn "we believe our model wasn't contaminated" into "here's the gap between public and private performance, measured."
What labs should publish #
If a lab reports a benchmark score, it should report the contamination controls alongside it: whether the run had network access, whether a held-out or post-cutoff split exists, and what the public-versus-private gap was. A score without those three facts is not wrong, exactly. It's just unreadable. You don't know which capability it measures.
And if the private split scores much lower, that's a finding worth publishing, not hiding. It tells everyone how much of the field's apparent progress on that benchmark was memory.
The prediction #
Benchmarks with public answer keys will saturate faster than benchmarks with private or rolling held-out sets, and the gap between the two will widen as training corpora grow. If public and private benchmarks keep rising at the same rate, contamination is a smaller effect than I'm claiming.
Until a lab shows me that gap, I read every public benchmark score as an upper bound on capability and a lower bound on memory.