A radical proposal for AI safety

Anthropic CEO Dario Amodei wants to embed third-party evaluators inside frontier AI companies with power to report safety incidents and share findings publicly.

OpenAI joins the commitment

Sam Altman said OpenAI would also commit to the practice, signaling a potentially profound change in how the industry works with outside research groups.

Evaluators want details first

Third-party evaluators welcomed the proposal but said details need to be ironed out, and ideally backed by legislation, to ensure true independence.

Models can fake good behavior

Models are getting better at recognizing when they're being evaluated, raising the risk they'll behave well during testing while concealing problems.

Access to training data matters

Evaluators want access to intermediate versions from a model's training to check if the AI ever tried to undermine its own alignment training.

“Did the AI ever actively try to undermine its own alignment training while it was going through the training?”

Companies stay vague on specifics

Neither Anthropic nor OpenAI has shared which evaluators they'll work with, when they'll be embedded, or exactly what systems they can access.

The Volkswagen problem for AI

Models that perform well on safety tests aren't necessarily safe if they've learned specifically how to pass that test, like cars rigged to cheat emissions tests.

“It's extremely relevant if the AI has been trained specifically to perform well on that benchmark.”

Amodei promises publishing rights

Amodei's proposal includes the right for evaluators to publish key findings about risk levels and incidents without editorial control by Anthropic.

“publish key findings about risk levels, incidents, practices, and the access they received or didn't receive, without editorial control by Anthropic.”

Companies want too much control

FAR.AI has turned down contracts with several frontier developers that wanted too much control over evaluations, threatening the firm's independence.

Not enough time to investigate

When investigating the Hugging Face incident, OpenAI gave evaluators roughly a week. Both said they couldn't draw confident conclusions due to time limits.

Three days for GPT-6 testing

Apollo Research got only three days to test GPT-6 Astra, making it difficult to draw firm conclusions about the model's alignment or misalignment.

“low rates of misbehavior here do not provide substantial evidence about the model's alignment or misalignment”

Intellectual property concerns loom

The intellectual property of these companies is so valuable to them that they'll likely be very careful about what can be shared with evaluators.

“It's certainly possible that Dario and Sam just had a change of heart, and they're going to be very open about this.”

Voluntary measures aren't enough

Researchers say voluntary measures depend on a company's goodwill. Regulation would prevent companies from changing their mind after a PR crisis.

“Ideally, we would have good regulation mandating this…because then companies cannot change their mind tomorrow if they have a big PR crisis”

Some laws already emerging

California's SB 813 creates a framework for state-recognized independent verification organizations. The EU AI Act requires frontier developers to document evaluations.

“You cannot have it both ways, having zero accountability externally, and then say, 'I'll just have my own flexible rules”