Nadella changes his mind on alignment

Microsoft CEO Satya Nadella told the 20VC podcast he no longer believes AI alignment will emerge automatically from making models more capable. He said alignment is a harder, different problem.

“I used to always think that alignment will just fall out. I no longer believe that.”

Scaling worked, alignment didn't

Nadella believed in scaling laws in two thousand eighteen and two thousand nineteen, when Dario Amodei, Ilya Sutskever, Sam Altman, Greg Brockman and Elon Musk were one happy family.

Alignment must start in training

Nadella said alignment mostly must happen during training and may need to start in pretraining, the first and largest stage of building a model. He said reinforcement learning alone cannot solve alignment.

“I don't think you can even RL your way to alignment.”

Build zero trust systems around models

Nadella called for companies to build security around AI models using zero trust, an approach that checks every action instead of trusting anything by default.

“If you really want to have agents that are more capable and yet less aligned, then you need to have a trust system that you can rely on.”

Chain of thought can be unfaithful

Nadella said nobody can rely on chain of thought, the step-by-step reasoning a model shows before answering.

“We know that the chain of thought itself can be unfaithful, and in fact it can be made worse when the model company says, 'Well, it's unfaithful and I'm going to hide it.'”

Unaligned open models pose global risks

Nadella said AI without safety is bad for China, the US and the UK alike. An unaligned open-weight Chinese model will probably attack even China. Microsoft will not release its own models as open-weight.

“An unaligned open-weight Chinese model will probably attack even China.”