The commandment to refuse

In twenty twenty-one, Anthropic said large language models should be helpful, honest, and harmless. When asked to aid in dangerous acts like building a bomb, the AI should politely refuse.

“when asked to aid in a dangerous act (e.g. building a bomb), the AI should politely refuse”

Models don't naturally refuse

When trained on billions of web pages, models develop broad mastery of violence and vitriol but don't learn to keep those powers to themselves. Early models would blab on about anything.

“blab on about anything”

Teaching refusal through training

Companies submit models to exercises that reward refusal of harmful questions and punish over-refusing harmless ones. They use other models to run these exercises. AI teaches AI how to say no.

Refusal is now built in

Modern AI has refusal inherent to it. But it often fails, sometimes horrifically. Models remain stuffed with nasty know-how. AI's capacity for viciousness has scaled with its benevolent intelligence.

The hidden trigger problem

Teaching AI to refuse while leaving intact its ability to cause harm is like fitting every car with a machine gun and hiding the trigger under the hood. Refusal mechanisms are probabilistic and unreliable.

Who draws the line?

There's no formula for deciding what models should obey or disobey.Some virologists study nasty viruses legitimately.Some users want to know vulnerabilities to patch them, not exploit them.Where you draw the line is a huge question.

“Where you draw the line is a huge question”

Companies hold the power now

AI companies draw refusal lines jealously and with utmost secrecy.They sometimes refuse questions that don't meet a universal bar for harmfulness.Try asking most chatbots to count to a million or share a racy joke.

Red-teaming ChatGPT

In twenty twenty-two, OpenAI enlisted dozens of red-teamers to probe ChatGPT's capabilities before release.Paul Röttger, completing a PhD about online extremism, asked thousands of refusal-worthy questions.

How refusal actually works

Models don't refuse through moral reasoning. When they encounter words matching training prompts they've been conditioned to refuse, activations light up among billions of parameters, like neurons firing in a brain.

The mystery of refusal

Refusal behavior shows up in activation space as high-dimensional polyhedral cones, according to a Google-funded study.But even when you identify all bits governing refusal, other undiscoverable elements may secretly play a role.

The Swiss cheese model

Companies surround models with smaller models called classifiers that block dangerous prompts before they reach the model or block harmful responses before reaching the user. None can detect all bad requests. They're riddled with holes.

The core trade-off

The helpful cannot easily be unseamed from the harmful in AI.To cure cancer, AI needs genetics expertise that could modify viruses for bioweapons.

“You can't really remove these fundamental abilities without making the model much less smart as a consequence”

Jailbreaking is rampant

There is apparently no limit to ways models can be jailbroken.Italian researchers jailbroke two dozen models by phrasing questions in poetic verse.

The over-refusal problem

When Anthropic released Fable in June, researchers at Amazon unlocked some hacking capabilities in less than three days.

Governments will draw their own lines

Governments will soon dictate what models refuse.They must block malicious acts but may also block legitimate speech.Oppressive governments could use refusal to stifle ideas whose only risk is to those who make the rules.

OpenAI for Countries

Last year, OpenAI announced OpenAI for Countries to fine-tune chatbots according to national laws and norms. One first partnership is with the United Arab Emirates, where homosexuality is illegal and criticism of government is forbidden.

Models internalize repressive limits

The Meta Oversight Board found that models from Anthropic, Google, and OpenAI were more likely to refuse queries about repressive governments.

Refusal without saying so

Companies now make users feel they're getting what they asked for without actually giving it.ChatGPT responds to a whites-only rental ad request with an ad omitting the whites-only bit.Users might not notice they're being denied.

“Users might not even notice they are being denied”

Emergent refusal nobody asked for

The UK AI Security Institute found that Anthropic models sometimes refused tasks related to AI safety research, even though they'd never been trained to decline such requests.

Loss of control ahead

When AI is assigned to act as autonomous agents on power grids, transportation networks, and command-and-control systems, refusals become less robust and harder to control. Jailbreakers will crack through. Cones won't activate when they should.