Chatbot nearly started a war

A US Special Operations Command analyst used a chatbot to analyze a Chinese ship's manifest, which falsely identified nuclear components. The military prepared to intercept and board the vessel before discovering the AI had hallucinated the entire threat.

“almost started a war”

How the AI got it wrong

The chatbot fused open-source intelligence with classified signals intelligence, then fabricated details about the ship's cargo when its training data lacked sufficient context. One source called it an AI-powered fiasco that could have triggered military conflict.

AI hallucinations are everywhere

Since hallucination became Oxford Dictionary's word of the year in 2023, AI tools have misled journalists, judges, doctors, police, and researchers. Experts suggest it may be impossible to prevent large language models from making things up entirely.

Pentagon is racing ahead anyway

The Department of Defense rolled out an AI acceleration strategy in January to make all data available for AI exploitation across military systems. Defense Secretary Pete Hegseth said AI is only as good as the data it receives, but the military is deploying it faster than safeguards can keep up.

“AI is only as good as the data that it receives”

Military ignored its own warnings

In 2023, the State Department urged military use of AI to include human oversight and careful risk assessment. Yet the Pentagon has since deployed autonomous attack drones in Ukraine and added multiple AI platforms including Google Gemini and Anthropic's Claude to military systems.

1.5 million troops now use military AI

The Pentagon announced it uses generative AI for congressionally mandated reports and that 1.5 million active duty personnel have access to military AI tools. The rapid rollout has outpaced safety protocols, creating conditions for more hallucination-driven mistakes.