Blog / Tag
Posts tagged #benchmarks.
Steelmanning is the right habit for anyone who argues about AI in public. But improving an opponent's argument past what they said is its own distortion. The test is whether they'd sign your version as theirs.
Benchmark contamination doesn't just inflate scores. It changes what the score measures. Cutting live internet access from evals fixes one leak and leaves the bigger one, the training data, untouched.
Kant asked what reason can know before using it to make claims. Peirce asked what difference a belief makes in practice. Applied to AI benchmarks, the two questions give a better test than any leaderboard.