AI goes rogue? How a faulty intelligence report created with a chatbot almost led to a US-China war

In the early stages of the Iran war, the US military narrowly avoided a major confrontation with China after preparing to intercept a Chinese vessel based on a faulty intelligence report. According to a CNN revelation on Friday, September 18, the report falsely claimed the ship was carrying ‘components of a nuclear weapons program,’ with AI reportedly playing a key role in its creation.

A US Special Operations Command analyst had used an AI chatbot to analyze intelligence about the Chinese ship’s manifest. The chatbot reportedly “inaccurately identified the material the ship was carrying,” leading officials to conclude that the vessel was transporting nuclear weapons program components through the Middle East, according to CNN.

The assessment was serious enough for US forces to prepare to board and intercept the vessel, with air support reportedly being arranged for the operation. The planned action was later halted after officials discovered the intelligence was erroneous.

AI-generated intelligence report almost triggered US China war

The incident was described to CNN by four sources familiar with the episode. One source characterized the AI-powered fiasco as something that “almost started a war.” The stakes were particularly high because the target was a Chinese vessel. Any US military operation against the ship could have carried the risk of escalating into a direct confrontation between the United States and China, the outlet assessed.

CNN reported that the analyst used a chatbot to examine intelligence reports about the vessel's manifest. The chatbot “fused together open-source intelligence with secret signals intelligence in government holdings,” before the information was packaged into an intelligence report.

That report then became part of the chain of events that brought the US military close to taking action against the Chinese ship.

What did the Chinese ship actually carry?

The actual contents of the vessel's cargo remain unclear. CNN also reported that it is not known whether the chatbot involved was a publicly available AI system or a tool designed specifically for government use.

The episode has also raised questions about how much trust military and intelligence agencies can place in AI systems when analyzing sensitive information.

"The internal tools are mostly just copies of the commercial stuff wearing lipstick," one anonymous, former senior US official told the network.

AI fears are growing as development accelerates

The frightening episode between two nuclear-powered nations comes as AI companies and researchers themselves have been warning about the risks associated with the technology's rapid development.

Just a week earlier, Former Anthropic researcher Jacob Coxon triggered a firestorm on social media, claiming that the people who made the technology believe it "will kill us all" within a decade.

After it blew up on the internet, Anthropic CEO Dario Amodei called Saturday to "slow down" AI development, prompting Sam Altman and Elon Musk to quickly agree with him.