From The Observation Deck · · 2m
The contagion of fear
A confessed college prank about a fake computer virus becomes the sharpest argument yet against AI doom talk.
In brief
A confessed college prank about a fake computer virus becomes the sharpest argument yet against AI doom talk. Experts borrow the public's trust when they speak, and fear spreads faster than the evidence for it, so extraordinary claims demand extraordinary proof. Originally reported by The Observation Deck.
A confession, decades late
Bryan Cantrill opens with a shameful story he has never told anyone, not even the people closest to him. It happened in his first year of university.
“I have a difficult confession to make.”
The prank in the lab
Late one night, arrogant CS underclassmen decided to walk next door to the general computer lab and tell humanities students a virus was loose on their machines.
Bedlam, not laughter
Students ejected floppies, killed the power, ripped cables out of dead machines, screamed, and fled. It was finals week, and people lost their term papers.
“the technological equivalent of shouting fire in a crowded theater”
Fear will not be recalled
The pranksters tried to walk it back. Some students responded with fury, others simply refused to believe the retraction. The fear had already taken root.
“once the fear had taken root, we couldn't eradicate it”
Letters in a drawer
The facilities director made each of them write two apology letters: one sent to the people they wronged, one kept in his desk as a standing threat. He returned it at graduation.
Why tell it now
He is confessing because he has never seen fear sown as irresponsibly by technologists as he sees now with AI and human extinction.
Greater than ten percent
Ex-Anthropic employee Jacob Coxon claimed, and Anthropic's Evan Hubinger agreed, that AI has a >10% chance of killing all humans within a decade.
“if you have a newborn, their claim is that there is a greater than ten percent chance that your child will somehow perish at the hands of AI before they enter middle school”
No mechanism offered
Asked how it would happen, the answers are hand-waves: hacked infrastructure, extinction-level bioweapons. Coxon is an expert in none of those fields.
Vector, not index case
Coxon caught the fear from others and passed it on. As more experts sound frightened, the head count itself becomes the evidence, and dissent gets drowned out.
“Falsehood flies, and the truth comes limping after it”
Intelligence is not enough
Building things takes character, hands, and reasoning about the physical world. Robots cannot do this work today, and are not on track to within the doom timeline.
“acts of engineering are not acts of intelligence alone”
The burden is on the claimant
Digital systems act in the world only where humans built and permitted that control. Extraordinary claims need extraordinary evidence, and the public should not have to supply it.
“domain experts, by way of their expertise, implicitly hold the public's trust, and we must not abuse it”
The short version
Experts borrow the public's trust when they speak, and fear spreads faster than the evidence for it, so extraordinary claims demand extraordinary proof.
