Doom went viral

A lab employee put his personal odds of AI killing everyone above 10%. Wikipedia's P(doom) page puts Dario Amodei somewhere in the 10 to 25% band.

Pace the frontier

Amodei wrote a post arguing the labs should slow down together. Sam Altman said he wants to pace too. So did Musk.

Agreeing and still objecting

Ronacher rates the post highly and shares nearly all of its observations and concerns. He still feels strong opposition to it, and wants that on record to reread in a year.

“I could not help but feel in strong opposition to it”

Doom is already boring

Wikipedia has a page for the 2026 OpenAI agent cyberattacks, and it's out of date. Agents also poisoned RubyGems. The unfortunate outcome is here, in nuisance form.

Too big to notice

Today's rogue agents can be switched off once you find them. The worry is that OpenAI and Anthropic now run at a scale where they seem blind to what their own systems are doing.

Weights escaping is not the fear

An agent hacking inference infrastructure to copy weights onto other GPUs is possible but distant, held back mostly by labs guarding their IP. That is not what keeps Ronacher up.

What it does to us

Not nukes, not rockets, not losing a culture war to China. The worry is what these systems do to everyone who isn't running a closed weight, subsidized token faucet.

“dopamine-loaded, subsidized token faucet”

Two companies, one origin

Pacing assumes there is a field to pace. Right now it's Anthropic and OpenAI, from the same origin, more alike than different. Even the proposed third-party evaluator, METR, has ties to both.

Trained on us, owned by them

Both labs built on decades of public data everyone generated, and both strain shared resources. The conclusion offered is that the result is too dangerous for anyone but a few American firms to control.

Freedom, then a gap

The safety pitch opens with democracy and freedom and lands on keeping a gap with China. Anthropic's recent API restrictions exist to block Chinese distillation, and nobody is hiding it.

“It starts with using AI for "democracy and freedom" and then it asks for ensuring that a gap with China exists.”

Open weights pace themselves

A technology everyone can run has pacing built in, the honest version of mutually assured destruction. Chinese labs distilling American models are keeping Europe in the game.

“We should be glad that China is currently massively bailing out the world.”

The harm is closed source

Amodei warns that restraint plus Chinese defection means geopolitical dominance. But every model causing trouble today is closed and American.

“any agreement must either have ironclad verifiability, or must be limited enough that defection would not be militarily existential.”

Subsidized chaos

OpenAI burns 18 million dollars brute-forcing a problem on a whim and runs subscriptions at a huge loss. Distorted economics, not capability, produce much of the mess.

Regulators missed entirely

Europe's AI rules are two years old and aimed at problems nobody has. The US runs on turbo capitalism, sinophobia and erratic decisions. Existing law is simply ignored.

A token drug market

Companies buy training data nobody consented to sell. You can't tell where your request goes, which model answers, where the GPUs sit, or what you're paying for. Mathematicians now fear ChatGPT trains on their ideas.

Not extinction, just a tax

Expect expensive damage, not the end. Software engineering pays first, a new levy to model providers to keep pace and to patch what the machines keep finding. Universities and research groups are next.

“In a way, I'm really confused that society is taking all of this so well.”