Sources: the US "almost started a war" after an AI-assisted report hallucinated that a Chinese ship in the Middle East was carrying nuclear weapons components (CNN)

In the midst of the war with Iran this spring, a US military intelligence report raised alarms when it falsely stated that a Chinese ship in the Middle East was transporting components of a nuclear weapons program. The report quickly prompted plans within the US military to intercept the vessel, according to four sources familiar with the episode.

Armed members of the military were reportedly preparing to board the ship, while military planes were in the air, ready for action. However, officials discovered that the report, generated by a special operations command analyst with the help of artificial intelligence (AI), was entirely false. The AI chatbot used by the analyst had inaccurately identified the cargo on the ship.

While the US military and intelligence community are increasingly integrating AI into various aspects of their work, this incident highlights the significant risks associated with relying on AI-generated information during wartime. The episode underscores the potential for catastrophic miscalculations when inaccurate or misleading data is used to target enemy vessels, which could escalate into armed conflict between nations.

Officials are pushing to further incorporate AI into military operations, from analyzing vast amounts of intelligence data to selecting targets for strikes. However, the decentralized nature of AI adoption across different parts of the government, along with the lack of standardized verification processes, raises concerns about the reliability and functionality of these systems.

The risks of AI in targeting have not gone unnoticed, with some analysts cautioning that the use of AI tools can lead to faster but potentially flawed decision-making, increasing the likelihood of civilian casualties or fratricide.