A lab can be perfectly transparent and still uncheckable. All it has to do is make checking expensive.
That's the failure mode I care about most in AI oversight, because it looks like openness. The paper is public. The methodology section is long. The eval suite is named. And yet nobody outside the lab can actually rerun the result, because doing so would take compute, access, or weeks of engineering that no independent party has.
The claim #
External verification is real only when its cost is within reach of the people expected to do it. A check that only the checked party can afford is self-assessment with extra steps.
Every oversight proposal should be evaluated by one number: what does it cost an outsider to catch a false claim?
How checking gets priced out #
There are several ways checking gets expensive, and most aren't malicious. They're just defaults that nobody bothered to change.
- Compute. Rerunning an eval on a frontier model, or reproducing a training-time result, can require resources only a handful of organizations have.
- Access. The model behind the published result is a snapshot that's no longer served, or served only with different safety layers, so outsiders test something adjacent to what was reported.
- Undocumented harness. The prompts, tool configuration, sampling settings, and scoring scripts live inside the lab. "We used benchmark X" is not reproducible without them.
- Time. By the time an outsider has rebuilt the setup, two newer models have shipped and nobody cares about the old claim.
Each of these raises the price of checking. Together they put it out of reach, while every individual disclosure looks generous.
Why "trust the auditors" doesn't fix it #
The usual answer is third-party audits. A lab hires or partners with an outside firm, gives it privileged access, and the firm reports.
That can help, but notice what it does to the cost structure. The outsider is now someone the lab chose, with access the lab granted, on a timeline the lab set. The check is cheap for that auditor because the lab subsidized it, and the lab can stop subsidizing it. That's not outside checking. It's inside checking with an outside letterhead.
I've predicted on X that lab-picked auditors with no deadline will file comfort letters before they file findings. The structural reason is this: an auditor whose access depends on the auditee's goodwill has a cost to finding problems that the auditee doesn't.
What cheap checking looks like #
Lowering the cost of checking is a design problem, and it has concrete answers:
- Ship the harness. Prompts, configs, scoring code, and seeds, in a repository someone can run with one command. This is often the single largest cost reduction available.
- Pin and preserve the model. Keep the exact evaluated version accessible to researchers for a stated period after the claim.
- Fund the checkers, not just the audit. Compute grants for independent replication, allocated by someone other than the lab, make checking affordable without making it captured.
- Publish small, decisive tests. Not every claim needs a full replication. A few targeted probes that would fail if the claim were false can be cheap and still bite.
- Put a clock on it. A result that hasn't been independently checked within a stated window should be labeled that way.
The common thread: shift cost from the outsider to the claimant. The party making the claim is the one who benefits from being believed. They should pay most of the price of being checkable.
The deeper point #
Science has always run on the idea that results should be reproducible in principle. AI has quietly let "in principle" do a lot of work. If reproducing a result requires resources nobody outside three companies has, then "in principle" means "not in practice," and the claim is resting on reputation.
Reputation is a fine prior. It is not a check.
What to watch #
The test I'd hold labs to: for any headline capability or safety claim, can a well-funded university group reproduce it within a month using only what was published? If the answer is regularly yes, the field is checkable. If the answer is regularly "it would take a year and a partnership," the field is running on trust, whatever its papers say.
I expect the second for now. I'd be happy to be wrong.