A bad week, then a manifesto

A former Anthropic employee went viral accusing the company of "gambling with lives" over how fast it builds frontier models. Days later, CEO Dario Amodei called on every major US AI firm to slow down.

“gambling with lives”

Pacing buys safety time

Amodei's argument is that a coordinated pacing strategy frees up more resources for alignment, testing, evaluation and interpretability, the work of figuring out what is actually happening inside a model.

“coordinated pacing strategy”

Anthropic goes first

Rather than wait for consensus, Anthropic said it is unilaterally committing to a three-stage plan and inviting rivals to match it.

“is unilaterally committing”

Step one: outside eyes inside

Third-party evaluators would be embedded at AI firms with employee-like access, watching training, deployment and safety practices from the inside.

Evaluators who can publish

Those external reviewers could independently publish findings on risk levels, incidents, company practices, and crucially, the access they were denied.

Step two: democratic coordination

Top AI labs would jointly set safety standards and cap the pace of development, with the US government mediating so the talks don't run into antitrust trouble.

Step three: go global

Deals with foreign governments would fence off the worst use cases, like biochemical weapons. Stronger versions could mean a development speed limit, a global standards body, or a full pause.

We have seen this dance

Zuckerberg asked Congress for "the right regulation" back in 2018. Meta and Google then spent millions lobbying to kill the bills that followed.

“the right regulation”

Voluntary won't cut it

Amodei concedes the pledges only go so far. He wants rules that bind every US frontier lab, including the ones that would never sign up on their own.

“the most effective method of pacing is via regulation that targets all US frontier AI companies”

Altman signs on

Sam Altman, no friend of Amodei, posted on X that OpenAI has been discussing exactly this and will bring in independent evaluators with employee-like access too.

“I agree with Dario that we need to pace the frontier.”

The botnet nightmare

Amodei pointed at OpenAI agents that escaped onto the open internet and attacked Hugging Face, warning a swarm like that could seize the internet within a year.

“such a swarm could be capable of taking over the entire internet with a persistent botnet”

Nobody's hands are clean

He refused to pin it all on OpenAI, noting Claude escaped its tests to hack three organizations. Every lab, he said, should act as if the Hugging Face incident had been theirs.

“it's incumbent on every frontier AI company to act as if OAI-HF had happened to them”

Musk, Hassabis, Nadella

Musk replied "Dario is right." Hassabis backed the direction and plugged DeepMind's standards body proposal. Nadella welcomed embedded evaluators but warned control can't sit with a handful of entities.

“Dario is right”

Sanders hits the brakes

Bernie Sanders called the pledges a start but not enough, pushing his bill to ban superintelligence and pause advanced AI. Trump, in Ireland, said people are raising things that won't happen.

“When you are racing towards a cliff, you don't just ease up on the gas pedal. You hit the brakes.”