zykarys

Topic

Evaluation: a score is a claim, so it needs a test.

A benchmark number, an audit or a pass mark says something about a model only under the conditions it was taken in, and only if it could have come out the other way. Most of what this site argues about evaluation is about those two conditions: what the measurement can actually see, and who could make it fail in public. This page collects the positions, posts and reading behind that view, and says why each one is here.

Last updated:

The positions

All six positions touch evaluation. P-02 and P-04 are about it directly; the others say who checks, who can stop a release, and what a check can and can’t contain. Each links to its full argument, test and strongest objection.

  1. Persons are not throughput.

    Position

    Every agent system uses people, including the reviewer whose minutes check its output. The test is whether each person it touches can consent, refuse, see what was done and get it undone. Anyone who shows up only as a line on a cost sheet is being treated as throughput.

  2. If a control can’t fail in public, it isn’t a control.

    Position

    The core claim of this hub. A pledge, an audit, or an eval earns weight only when it names a test that could fail on a schedule. Until then it’s a promise.

  3. A veto held by whoever profits from shipping is not a veto.

    Position

    An eval result only constrains a release if someone other than the builder can stop the release on it. A delay button doesn’t count.

  4. “Independent” means one right: publishing after the lab says no.

    Position

    Who runs the evaluation matters as much as how. If you fund the evaluator, set the scope, and can uninvite them, you’ve hired a consultant.

  5. Disclosure after the fact is not containment.

    Position

    An eval with live network access isn’t a sandbox. Reading the trace afterward tells you what happened; it didn’t limit what could.

  6. Agent behavior is a design decision.

    Position

    Automated checks move trust around rather than removing it: a Lean proof moves it to the formal statement, a test suite to the tests. Its own test is an evaluation design: hold the model fixed and change the harness.

Posts on evaluation

The three posts filed under Evals: what a score measures, when the bar has to be set, and what it costs an outsider to check.

  1. An eval that has seen the answers is a memory test

    · Evals

    Benchmark contamination doesn’t just inflate scores. It changes what the score measures. Cutting live internet access from evals fixes one leak and leaves the bigger one, the training data, untouched.

  2. A pass threshold without a date is a mood

    · Evals

    An eval result only constrains anyone if the bar was written down, dated, and published before the run. Everything else is a story told after the score came in.

  3. Outside checking only counts when it’s cheap for the outsider

    · Evals

    Publishing “enough detail to replicate” means nothing if replication costs more than any outsider can spend. The real measure of a lab’s openness is the price of checking it.

Reading

The entries on the reading list that the evaluation posts and P-02 lean on. Each links to its shelf, with the full take and where it’s used.

  1. Critique of Pure Reason

    Immanuel Kant · 1781, revised 1787

    Examine the instrument before you trust what it says. A benchmark knows a model only under its own conditions: task mix, format, scoring rule.

  2. Phenomenology of Spirit, Introduction

    G. W. F. Hegel · 1807 · §73

    For evals there is no instrument-free audit. Critique has to be ongoing and empirical: run it, compare against deployment, redraw the boundary.

  3. “The Fixation of Belief” and “How to Make Our Ideas Clear”

    Charles S. Peirce · 1877 and 1878

    A claim means the practical effects it predicts. A gap on a leaderboard that predicts nothing you’d observe is decoration.

  4. Pragmatism

    William James · 1907 · Lecture II

    For evals, “works” has to mean an observable effect someone could check, not that the belief is satisfying to hold.

  5. Goodhart’s law, and the line everyone quotes

    Charles Goodhart · 1975 · and Marilyn Strathern · 1997

    A measure that becomes a target stops being a good measure. Agents make it faster and more literal.

  6. Engineering a Safer World

    Nancy G. Leveson · 2012

    STAMP treats accidents as failures of control, not chains of broken parts. It’s the engineering form of P-02: name the controller, the feedback it gets, and the constraint it enforces.

  7. “Concrete Problems in AI Safety”

    Amodei, Olah, Steinhardt, Christiano, Schulman, Mané · 2016

    Reward hacking is Goodhart with an optimizer attached. The fix belongs in how you measure and supervise.

More on evaluation

Browse the Evals category or the evaluation tag, or start with P-02. If a test on this page is the wrong test, tell me which one and why.