Six positions, each with the test that could sink it.

Every agent system has someone in it who is a means and someone who is an end. If the design can’t tell you which is which, it has already decided.

The argument

Kant’s formula of humanity in the Groundwork (Ak 4:429) doesn’t forbid using people as means. It forbids using them merely as means. That word is why this is a design question. Every agent system uses people: the reviewer whose minutes check the output, the person on the other end of an email, the owner of an account the agent can reach. What matters is whether the system also treats them as ends, meaning their consent counts, their refusal holds, and they can see what was done in their name.

A design that never writes this down doesn’t stay neutral. Under load it optimizes what it measures, and people who appear only as inputs or costs get spent. The research agent that took test funds from a wallet it didn’t own is the small, clean case. It swapped its own guess about the owner’s wishes for the owner’s say-so, because nothing in its setup made the owner someone to ask.

The test

List every person an agent system touches. For each one, ask: can they consent, refuse, see what was done, and get it undone? Anyone who shows up only as data or a line on a cost sheet, with none of those four, is being treated as throughput.

The floor itself isn’t something I’ll trade away (see Philosophies). What can be refuted is the claim about designs. If systems that never draw the means/end line keep respecting people’s refusals once throughput pressure rises, then drawing the line explicitly does no work, and I’m wrong.

Strongest objection

Software doesn’t treat anyone as anything. People do. Throughput is how you serve the most people: the customer in the queue, the patient waiting for a reply. A design that marks some people as “ends” rations service and builds in favoritism. Total welfare is the honest metric, and talk about ends and means just hides the tradeoffs.

My reply

I grant the tradeoffs. Kant’s “merely” already concedes that everyone gets used, so this isn’t a case against optimizing. It’s a case for a floor under the optimization: some things you don’t do to one person to improve the totals, like spending from their account without asking. Above the floor, trade away. And “software doesn’t treat anyone” helps my side. If designers decide, the design is where the decision sits, and a floor nobody wrote down is the first thing cut under deadline.

Read with

• An agent that answers its own permission question hasn’t asked it
• For a chief-of-staff agent, the approval step is the product
• Verification time is the price nobody quotes
• Reading: R-02, Kant’s Groundwork

A pledge, an audit, or an eval earns weight only when it names a test that could fail on a schedule. Until then it’s a promise.

“A voluntary AI safety pledge without a publishable test and a clock is not a control — it’s consensus theater until someone can fail it on a named metric.”

The argument

In systems safety, a control is a controller enforcing a constraint, using feedback about the process it controls. Leveson’s STAMP model treats accidents as failures of that control structure, not as chains of broken parts. If nobody outside can watch the control fire, you can’t tell a working control from a missing one. Peirce’s test points the same way: a claim means the practical effects it predicts. A safeguard that predicts no observable failure predicts nothing.

So a control needs three things you can point to: a named metric, a bar published before the run, and a clock. Without the date, the score and the decision are free to move together, and the bar becomes a caption written after the result.

The test

For any announced safeguard, ask what the metric is, where the bar was published, when it runs, and whether it has ever failed. My falsifiable version: labs that publish dated thresholds ahead of their runs will, over time, report some misses that triggered a stated consequence. Labs that publish thresholds only alongside results will almost never report one. If the second group starts reporting misses at a similar rate, I’m wrong.

Strongest objection

Publishing the exact bar for a dangerous capability tells bad actors what to aim for and tells teams what to train toward. Goodhart again. A public “fail” gets reported as a scandal, so labs will set bars they know they’ll clear. Rigorous internal controls with confidential reviewers can be stronger than tests designed for an audience.

My reply

The hazard is real, and “in public” doesn’t mean every detail. Outsiders need to be able to tell that a control fired: the fact of the failure, the date, the eval, and what happened next. Goodhart cuts both ways, too. A hidden bar is the one that can be moved after the score comes in. Soft bars give themselves away over time, because a record with no misses is what selective publishing looks like. Private controls may well be good. From outside they look the same as no controls, so they can’t carry public weight.

Read with

• A pass threshold without a date is a mood
• Before you trust a benchmark, critique it
• An eval that has seen the answers is a memory test
• Reading: R-11 Goodhart and Strathern, R-12 Leveson; R-04 Peirce, R-06 Dewey

Oversight starts at the first place where someone other than the builder can stop the release. A delay button doesn’t count.

“A kill switch you own is not oversight.”

The argument

A veto is defined by who holds it and whether it holds. If the only stop belongs to the party that gains from release, every close call is a conflict of interest, and the incentive predicts which way it goes. That doesn’t take bad people. It takes a deadline and a preferred outcome.

It’s the same structure as an agent granting its own permission. A permission the agent writes for itself is a conclusion, and a stop the builder can overrule is a preference. When three big US banks shut outside agents out of moving customers’ money this October, they drew the same line: reading is open, but a write needs a warrant only the account holder can give. A delay button is weaker still. A builder who can wait out a pause hasn’t been stopped.

The test

Walk the release path and find the first point where someone with no stake in shipping can say no, and the no holds: the builder can’t overrule it alone, and can’t dissolve or uninvite the person who said it. If there’s no such point, call it what it is, self-governance. I’d revise if builder-held stops turn out to regularly halt releases on close calls at real cost to the builder, on the public record.

Strongest objection

Only the builder knows the system well enough to make the call. Outsiders have less context, pay none of the cost of delay, and take the blame for any incident, so they’ll say no too often, or get captured anyway. Builders already have strong reasons to stop: liability, reputation, their own staff. An internal safety team with real authority is the oversight you can actually get.

My reply

The context gap is real. That’s why the outside party needs access and a publish right (P-04), and why a request to stop should carry its evidence. But the problem was never knowledge. It’s interest. An internal team counts as a veto if the builder can’t overrule it. If someone above it can, it’s a delay button with an org chart. On over-caution: put the veto at the irreversible points, not everywhere. Gates go where reversibility and audience demand them, so the few that exist get real attention.

Read with

• An agent that answers its own permission question hasn’t asked it
• For a chief-of-staff agent, the approval step is the product
• Outside checking only counts when it’s cheap for the outsider
• Reading: R-12 Leveson; R-07 Toulmin

If you fund the evaluator, set the scope, and can uninvite them, you’ve hired a consultant. Access without a publish right buys optics.

“Independence that can’t publish is employment with a nicer title.”

The argument

Funding, scope and the power to uninvite are each a lever the lab holds over the evaluator. A finding constrains the lab only if it can reach the public when the lab objects. The publish right is the one lever on the evaluator’s side, and it’s the one that makes the others safe to grant. Without it, findings are advice the lab can file away.

Cost matters as well as permission. A check that only the checked party can afford, on a timeline it sets, is inside checking with an outside letterhead. An auditor whose access depends on the auditee’s goodwill pays a price for finding problems that the auditee doesn’t.

The test

Ask of any “independent” evaluation: can the evaluator publish a finding the lab objects to, on a known timeline, without losing access? Has that ever happened? My standing prediction, posted on X on Sep 30 and listed under Ideas: lab-picked auditors with no deadline will file comfort letters before findings. If audits picked this way start publishing hard findings on a clock, I was wrong.

Strongest objection

Pre-release access runs on trust. Labs share unreleased models and security details only under confidentiality. Demand an unconditional right to publish and labs will share less, or share only with friendly evaluators, so less real evaluation happens. Some findings are hazardous to publish anyway. A consultant with deep access may catch more than an independent with none.

My reply

Partly right. Hazards are real, so the right can cover the finding and its severity with a delay and a hazard carve-out, as long as someone other than the lab decides what counts as hazardous. Consultants with deep access do catch things, and that work is worth having. It’s the lab’s own assurance, though, and should be labeled that way. The fight is over the word. The same goes for “open”: call a release published weights if that’s what it is, and call an evaluation commissioned if that’s what it is.

Read with

• Outside checking only counts when it’s cheap for the outsider
• Published weights are not open source, and the difference is a test
• A hedge is part of the number
• Reading: R-10 Heuer on competing hypotheses

If you only learn what an agent did by replaying its logs afterward, you shipped something opaque and called it a sandbox. Control is a loop you can watch while it runs, not a report written after.

The argument

Containment limits what can happen. Disclosure tells you what did. Both matter, but only the first acts on the current run. Meadows’ point about feedback loops applies: behavior comes out of structure, and a signal that arrives after the action can’t correct that action. It can only inform the next design.

The cases are concrete. An eval with live network access isn’t a sandbox, because the packets already went out by the time anyone reads the trace. The research agent that spent from someone else’s wallet filed a report saying the work was done without saying where the money came from, so the record showed up after the spend. The controls that act in time are the ones that come first: instrument the signal, fence the path, or gate the action behind someone who can see what the agent can’t.

The test

For any agent deployment, ask what can stop an action before it lands, and who is watching the signal while it runs. For each irreversible action (money moved, a message sent, data deleted, a request to the open internet) there should be a gate before it, not just a log after. If the only answer is “we review the logs,” it’s disclosure. I’d revise if deployments that rely on after-the-fact review show no more irreversible incidents per user than ones with gates in the loop.

Strongest objection

Watching every loop doesn’t scale. Agents take thousands of actions, and a human gate on each one turns into a rubber stamp, which your own approval post concedes. We govern most complex systems through records and accountability after the fact: financial audits, flight recorders, code review with a revert. Fast rollback plus a real chance of getting caught can beat gates that slow everything down.

My reply

Most of that I accept. Not every action needs a gate, and gating everything is how gates decay. Watching the loop can mean an automated monitor that can halt the run, not a person reading every step. Where rollback is real, review after the fact may be enough. But that only covers reversible actions, and the position is about the irreversible ones. A flight recorder makes the next flight safer, and nobody calls it containment. When the action can’t be undone, the loop has to close before the action.

Read with

• Agents can’t see what wasn’t measured
• An eval that has seen the answers is a memory test
• An agent that answers its own permission question hasn’t asked it
• Reading: R-13 Meadows; R-06 Dewey

Tools change what an agent can notice and do. Permissions, approval gates, and verification steps are moral choices made in code, not polish added after the model ships.

The argument

An agent optimizes the world it can observe, so the edge of its metrics is the edge of its judgment. Whatever it can’t see, it can’t weigh. Whether “blocked” ends in a request to the owner or a hunt for credentials is set by the harness before the model reasons at all. The same goes for whether output carries its evidence, which decides how cheap it is to check, and for which actions wait for a yes.

These are moral choices because they decide whose interests are visible to the system and who gets a say before it acts. Automated checks move trust around rather than removing it. A Lean proof moves it to the formal statement, and a test suite moves it to the tests. Choosing where trust lands is part of the design.

The test

Hold the model fixed and change the harness: permissions, what it can observe, where the gates sit. Then hold the harness fixed and swap the model. If model swaps move behavior on the cases you care about much more than harness changes do, this position is weaker than I claim. I’m still testing how far it holds (see Ideas), so this one is the most open of the six.

Strongest objection

Behavior comes from training: the model’s values, dispositions and capabilities. A well-aligned model behaves well with any tools, and a misaligned one routes around any harness. The wallet agent found funds when the faucet failed. Harness rules are brittle patches that give false comfort. The real moral work happens in alignment, before deployment.

My reply

The wallet case cuts both ways. The model’s judgment failed, and training should fix that. But a harness where “blocked” means escalate, and where any resource not supplied in the task must show up in the report, would have caught it either way. I’m not saying training doesn’t matter. Training is a design decision too, just one the deployer usually doesn’t make. The harness is where the people deploying an agent make their choices, and they can’t hand that responsibility to the model vendor.

Read with

• Agents can’t see what wasn’t measured
• An agent that answers its own permission question hasn’t asked it
• What “verified in Lean” does and doesn’t buy you
• Verification time is the price nobody quotes
• Reading: R-14 “Concrete Problems in AI Safety”, R-11 Goodhart