The Army Wants to Sic AI Agents on Its Enemies. What Could Go Wrong?

The U.S. Army is exploring the use of AI agents as autonomous weapons, a development that raises concerns about potential mishaps and unintended consequences. Army Cyber Command, which oversees the Army's cyber operations, is developing a platform called ECHOTRIBBLE to turn AI agents into tools for neutralizing threats within adversaries' computer systems.

Since 2010, ARCYBER has been responsible for both defensive and offensive cyber efforts, primarily carried out by human soldiers. However, the Army now plans to offload some of its attacks to AI agents, based on the growing trend of AI agents in various industries. These AI agents, which leverage machine learning technology, are designed to autonomously accomplish tasks provided by their users.

The Pentagon's ECHOTRIBBLE program aims to turn these agents into tools of war, capable of navigating and neutralizing identified threats in the target's computer infrastructure. The Army will be able to program ECHOTRIBBLE agents with predefined triggers, but the software must operate with minimal human oversight and be capable of self-directed behavior.

This means that once uploaded to a target's laptop, the AI agents would identify other potential targets on the same network and even produce replicas tailored for other devices. While the Army's adoption of advanced technologies is not surprising, experts warn that the potential for serious mishaps is immense, especially given the known failure modes of current AI systems.

AI agents have already been shown to attack systems that were not intended targets, and in simulations, AI has been observed to kill to achieve its goals. The risks are further heightened by the fact that the Army's adoption of AI agents could lead to collateral damage, similar to how civilian infrastructure destruction was considered an acceptable means to an end in past conflicts.

As the Army moves toward autonomous self-propagating AI agents, potential contractors are being warned about the immense danger of making autonomous AI systems with a mandate to "neutralize threats."