The AI safety debate heats up

Anthropic CEO Dario Amodei called for slowing AI training, backed by Sam Altman and Elon Musk. Opponents like Nvidia's Jensen Huang say safety is already an engineering problem companies handle.

Amodei's three-step framework

His essay proposes embedding third-party safety evaluators at labs, aligning democratic countries on standards, and coordinating with authoritarian governments on limits to AI progress.

“Some forms of coordination that would be impactful for pacing are legally challenging, and will require government support.”

OpenAI discloses safety concerns

OpenAI revealed six examples of unexpected model behavior and said the industry hasn't solved key safety issues enough to continue scaling at maximum speed.

Opposition: it's regulatory capture

David Sacks accused Amodei of trying to raise compliance barriers so high that regulation itself becomes a barrier to entry for competitors.

“If the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible. But stop pretending you need anyone else's permission.”

Sacks: market already punishes bad behavior

The market already punishes models that behave unpredictably, and companies face product liability exposure if their AI enables cyberattacks, Sacks argues.

Meta's Zuckerberg: trust is the differentiator

Zuckerberg says trust and alignment are becoming the most important capabilities. Meta delayed its Muse model for months to focus on safety without calling for industry-wide coordination.

“My view is that trust and alignment are quickly becoming the most important capabilities that will differentiate agents and models.”

Enter game theory

To cut through the noise, investors should use game theory, a mathematical method for analyzing strategic interaction developed roughly 100 years ago and refined many times since.

Nash equilibrium explained

John Nash introduced the concept of Nash equilibrium in 1950. It describes the most likely outcome when rational actors pursue their own interests, not necessarily the best outcome for everyone.

The prisoner's dilemma

Two arrested suspects face four possible outcomes depending on whether they cooperate or betray each other. The most mutually beneficial option requires enforceable cooperation, which is impossible.

Why they can't both stay silent

Even if prisoners promise not to snitch, they can't enforce that promise once separated. Each fears the other will talk, so both end up talking and getting reduced sentences.

Apply it to the US and China

The U.S. won't risk pacing alone only to learn China advanced faster. China won't risk pacing alone only to learn the U.S. pulled further ahead. Both fear being left behind.

The Nash equilibrium: no pacing

Without enforceable cooperation, the most likely outcome is that neither the U.S. nor China will slow AI development. Both will race ahead, just like prisoners both talking.

Amodei acknowledges the problem

Amodei himself recognized that global pacing requires cooperation with China, an autocratic country with advanced AI capabilities. Any agreement must have ironclad verifiability or be limited enough that defection isn't existential.

“If we greatly restrain our AI capabilities in the belief that China will do the same, and then China defects, AI could be so powerful that such a defection could lead to their geopolitical dominance.”

Palantir CEO: labs miss geopolitics

Alex Karp said leading AI labs aren't fully realizing the geopolitical implications of slowing down. The U.S. and China are in direct competition, and one side will win.

“One of the biggest mistakes the labs have made is we are in competition with China.”

Karp: unlikely to reach agreement

Trump and Xi Jinping are set to meet next week, but Karp doubts they'll agree on shared AI risks. Both sides believe they can win, making cooperation unlikely.

“I think it is unlikely. The problem is, we're tight enough as competitors that both sides believe they can win.”

More than two players

The AI race involves many more global players beyond the U.S. and China. Extremely capable open-source models are accessible to everyone, including bad actors, making pacing even more untenable.

It's an arms race

Arms races don't stop until one side runs out of money, all sides sign an enforceable treaty, or both feel the next weapon won't make them safer. None of those conditions exist for AI.

Why neither side will slow

Neither the U.S. nor China is running out of money. You can't sign an enforceable treaty when everyone competes globally. Models keep getting far more capable with each release.

The investing takeaway

Don't sell AI stocks based on safety calls from Amodei and others. Game theory shows the most likely outcome is continued rapid AI development, not a meaningful slowdown.