A pass threshold without a date is a mood
An eval result only constrains anyone if the bar was written down, dated, and published before the run. Everything else is a story told after the score came in.
Blog / Category
Evals
Posts filed under Evals.
An eval that has seen the answers is a memory test
Benchmark contamination doesn't just inflate scores. It changes what the score measures. Cutting live internet access from evals fixes one leak and leaves the bigger one, the training data, untouched.
Outside checking only counts when it's cheap for the outsider
Publishing "enough detail to replicate" means nothing if replication costs more than any outsider can spend. The real measure of a lab's openness is the price of checking it.