A benchmark score is a claim about what a model can do. Before accepting that claim, it's worth asking what the benchmark could possibly know — and what difference believing it would make. The first question is Kant's. The second is Peirce's. Together they're a better filter than a leaderboard.
Kant's move: examine the instrument first #
Kant's Critique of Pure Reason (1781, revised 1787) begins from a worry. Metaphysics kept producing grand claims that contradicted each other, and nobody had asked whether reason was equipped to settle those questions at all. So Kant proposed to examine the faculty before trusting its outputs: to map what it can know, under what conditions, and where it overreaches.
His answer, compressed: we know things as they appear to us, structured by the forms of our experience and our concepts. We don't know things as they are in themselves, beyond any possible experience. Claims that go past the limits of possible experience aren't false so much as ungrounded.
Translate that to evals. A benchmark is an instrument for knowing about a model. Like Kant's reason, it has conditions: a task distribution, a format, a scoring rule, a sampling setup. It knows the model as it appears under those conditions. It doesn't know the model in itself — its behavior on every task, in every deployment, under every prompt.
The overreach is the same one Kant diagnosed. A score earned under narrow conditions gets cited as if it measured a general capability: "reasoning," "coding," "safety." That's the benchmark claiming knowledge beyond the limits of the experience it can have.
What a critique of a benchmark asks #
Kant-style critique isn't skepticism. It's boundary-drawing. For a benchmark, it asks:
- What conditions shape the appearance? Format, prompt template, tool access, number of attempts, scoring rubric.
- What's outside the possible experience? Tasks unlike the distribution, longer horizons, adversarial users, real deployment context.
- Which claims stay inside? "Scores N on this suite under these settings" stays inside. "Is good at reasoning" goes outside.
Notice that contamination is a Kantian problem too. A model that memorized the test makes the appearance unreliable even inside the boundary, because the instrument's conditions have been breached.
Hegel's objection, and why it doesn't bite here #
Hegel had a famous complaint about Kant's project: insisting on examining knowledge before knowing anything is like refusing to enter the water until you've learned to swim. You can only test an instrument by using it.
Fair, and it applies to benchmarks. You can't critique an eval in the abstract; you learn its limits by running it and seeing where it misleads. But that supports the point rather than defeating it. The critique should be ongoing and empirical — run, compare to deployment, find the gap, redraw the boundary — not a one-time blessing.
Peirce's move: what difference does it make? #
Charles Sanders Peirce, in "How to Make Our Ideas Clear" (1878), offered a test for meaning: consider what practical effects the object of a conception could have, and that is the whole of your conception of it. William James, popularizing the idea in Pragmatism (1907), put it as: there can be no difference anywhere that doesn't make a difference elsewhere.1
Apply it to a benchmark claim. "Model A beats Model B on this benchmark" means exactly what it predicts about practice. If the score difference predicts that A will do better on the work you care about, the claim has meaning for you. If it predicts nothing you'd ever observe — because the benchmark tasks don't resemble your tasks — then for you the claim is empty, however precise the number.
That gives a pragmatist definition of a good benchmark: one whose scores predict observable differences in the uses people actually make of models. A benchmark that ranks models in an order that deployment never reproduces is, in Peirce's sense, not saying anything.
Where the two meet #
Kant tells you where a benchmark's knowledge stops. Peirce tells you whether what's inside that boundary matters. Put together, they give two questions I ask of every score:
- Under what conditions was this observed, and is the claim staying inside them?
- What would I expect to see differently in practice if this claim were true?
If the first answer is "unclear" or the second is "nothing," the score is decoration.
There's a tension I don't want to hide. Kant held that some things bind us whether or not they pay off. Pragmatism says keep what works. For benchmarks I lean pragmatist: a test earns trust by predicting. For the rules around testing — don't move the bar after the run, don't hide failed results — I lean Kantian. Those hold even when bending them would work out fine this once.
What to watch #
Benchmarks validated against deployment outcomes will keep their influence. Benchmarks that only correlate with each other will be quietly abandoned as soon as they saturate. If a benchmark with no demonstrated link to real use stays central for years, prestige beat prediction, and pragmatism lost that round.
Notes
- James, Pragmatism, Lecture II, "What Pragmatism Means." The same lecture credits Peirce's 1878 article in Popular Science Monthly as the origin of the principle. ↩