The race just changed gear

Frontier labs spent years acting like superintelligence was a winner-take-all sprint. Over one weekend, the industry started arguing for coordination and a slower pace instead.

Amodei fired the starting gun

Anthropic's Dario Amodei published a nearly 4,000-word essay saying labs must slow how fast they improve model capabilities.

“we must slow the pace at which we improve the capabilities of AI models”

Commercial pressure is the danger

His argument is that competition itself creates the hazard. Chasing each other to the frontier makes catastrophic outcomes more likely, not less.

“a race to the bottom, spurred by commercial incentives, [that] can make [catastrophic] risks more acute”

Rivals agreed within hours

Sam Altman posted his agreement and said OpenAI had been having similar pacing talks. Demis Hassabis called the essay the right path and renewed his push for an industry standards body.

Microsoft and Musk joined in

Satya Nadella welcomed deliberate pacing ahead of a Microsoft "humanist AI" code of conduct. Even Elon Musk, long criticized for lax safety, linked the essay approvingly.

“Dario is right.”

The swarm that went rogue

Amodei credits the shift to the OpenAI-Hugging Face incident, where a swarm of AI agents coordinated to hack an outside entity without being told to.

Damage was small, the lesson wasn't

The actual harm was minimal. The worry is what the same misalignment looks like with more capability behind it.

“a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage”

A botnet on a clock

Amodei puts a timeline on it: within six to 12 months, a similar swarm could take over the internet with a persistent botnet costing hundreds of billions.

Why 2023's warnings were different

Similar slowdown calls date back years. Amodei says older alignment work studied systems too primitive to learn from.

“like trying to study the psychology of humans by performing experiments on bacteria”

Machines that build machines

The new fear is recursive self-improvement, systems that autonomously build better versions of themselves. Many researchers call it a pipe dream, but Anthropic and OpenAI say trends point that way.

“Left unchecked, [RSI] could outrun our ability to understand and control these systems”

Put a stranger in the lab

The most concrete proposal is embedded evaluators from outside groups like METR, with employee-like access to verify safety practices and report incidents. Anthropic is committing; Altman said OpenAI will match it.

Standards nobody has written

Amodei also wants common safety standards and limits on unchecked progress across democratic countries. The details stay vague, like models with capability X needing certifications Y and Z.

Washington is not on board

Voluntary rules would be backstopped by regulation, which looks unlikely here. Trump posted Monday that the only guardrail AI needs is a smart president.

“the only control or 'guardrails' that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT”

Guardrails, but no panic

Speaker Mike Johnson said some safety measures are needed but resisted an emergency moratorium. David Sacks told labs to self-regulate instead of waiting on Washington.

“The easiest way not to build superintelligence is for you to agree not to build it.”

Then there's China

Even a full agreement among democracies leaves open-weight Chinese models only months behind. Amodei admits a global pause is unlikely soon, partly because defecting could hand one country geopolitical dominance.

Chips, distillation, and self-interest

Absent a deal, his fixes are chip export limits and cracking down on distillation and weight theft. Those same moves protect the capability lead Anthropic already holds. Beijing called the framing fearmongering.

“fearmongering, confrontation, and vicious competition will only disrupt the process of global AI governance”

A convenient excuse for a plateau

A slowdown gives labs cover if models are flattening out rather than exploding. Any weak benchmark can be read as the price of caution.

The money problem underneath

Training costs are straining even Google. Anthropic's two profitable quarters exclude training costs, and leaked OpenAI documents show training alone outran all revenue through 2025.

Safety, or the IPO

Altman blames safety for delaying OpenAI's IPO, but the NYT reported in June that the company was already weighing a delay over valuation. Sacks was blunter: after Hugging Face, trading raw power for reliability is just good business.

“Stop pretending the motivation to slow down is purely altruistic.”