AI labs want embedded watchdogs

Anthropic CEO Dario Amodei proposed third-party safety evaluators inside frontier AI companies with access to closely guarded systems.

Bank supervisors as the model

Amodei explicitly cited embedded bank supervisors as precedent, promising evaluators access comparable to internal risk teams.

But they lack enforcement power

Unlike bank supervisors who can close institutions, the proposed AI evaluators have no authority to halt model training or release.

“If you don't give them that kind of power, I don't know what they are doing”

What bank regulators actually do

Government examiners at large banks can stop practices, restrict growth, force management changes, and close institutions.

The comparison doesn't hold up

Banking expert Julie Andersen Hill says without kill switch power, the banking industry comparison isn't accurate.

“That's fundamentally different, because bank regulators have a lot more power than that”

A researcher's warning sparked this

Former Anthropic researcher Jacob Coxon resigned, warning that frontier labs were racing toward systems they might not control.

What evaluators see in practice

Albert Ziegler at XBOW evaluates unreleased models from major labs. He finds formatting bugs, not catastrophic AI behavior.

“But the kind of insidious, catastrophic consequences produced by subterfuge combined with unprecedented abilities that people are afraid of, that's not something we've seen ourselves”

Testing has serious limitations

Black-box testing can reveal dangerous capabilities, but serious risks may only emerge under rare combinations evaluators never trigger.

No veto power over releases

Evaluators can document risks and compel informed decisions, but the ultimate choice to release a model stays with the company.

“It's true that we don't have any veto power”

Even bank supervision fails sometimes

Bank supervisors have failed to prevent major collapses and are regularly accused of becoming too close to the institutions they oversee.

Looks like internal compliance

If companies select evaluators, control access, and ignore conclusions, it resembles an internal compliance department, not regulation.

“If Anthropic wants an internal compliance department, there's nothing currently stopping them from having one”

METR's conflict of interest problem

Anthropic named nonprofit METR as a potential evaluator, but the organization has worked with Anthropic and recently hired a former researcher.

Independence requires more than access

UC Berkeley researcher Deborah Raji says true auditors must meet competence requirements and rules governing conflicts of interest.

“You can't just wake up one day and decide that you're qualified to be a bank examiner”

The field is too interconnected

METR employees have strong social ties to AI company workers and share research space with lab employees.

“They're suspected of financial and ideological entanglement, personal conflict of interest”

Companies control the scope

In METR's OpenAI contract, the company set audit parameters, controlled access, and decided which findings could be published.

AI auditing needs new expertise

Traditional audits check processes and controls. AI auditors must verify the actual system and its output, requiring rare technical skills.

No legal framework exists yet

Frontier AI has no body of rules defining what's prohibited, where evaluators have discretion, or what follows a serious finding.

“It doesn't really work to just turn supervisors loose without any standards to hold them to”

Audit washing without consequences

The goal of an audit is consequential judgment. If you do an audit and nothing happens, that's audit washing.

“If you do an audit and nothing happens, that's audit washing”

Can't have control and credibility

Access and publication rights produce scrutiny, but can't confer regulatory credibility while companies retain control over outcomes.

“You can't have all of the control and then expect the credibility as if you've given up control”

The ultimate test of seriousness

If AI truly has power to destroy society, you need an independent supervisor with ability to pull the plug.

“If you really believe that AI has the power to destroy society, then you have to have an independent supervisor that has the ability to pull the plug on it”