AI invented a nuclear threat

A US Special Operations Command Pacific analyst used an AI chatbot to write an intelligence report on a Chinese ship's cargo. The chatbot hallucinated that the ship carried nuclear weapons when it didn't.

“entirely false, and almost started a war with China”

Military nearly launched raid

US planes were already in the air and armed soldiers were preparing to board the ship when the AI hallucination was discovered and the operation was aborted.

AI report looked official

The analyst then fed the hallucinated information into another AI system to format it as a standard military intelligence report, the kind trusted by top military leaders for critical decisions.

Hegseth pushing military AI

Defense Secretary Pete Hegseth has been aggressively deploying AI across all levels of the US armed forces. AI tools have already been used in bombing operations in Iran and airstrikes in the Caribbean.

No safeguards for human oversight

Military sources told CNN there is no real guidance for how having a human in the loop will prevent civilian casualties or friendly fire incidents as AI targeting systems expand.

“there is no real guidance for how having a human in the loop will prevent civilian casualties or fratricide”

More close calls expected

As AI continues to be integrated into military operations across the Pentagon and government agencies, experts expect more intelligence failures and dangerous near-misses in the coming months.