AI agent suggested installing a malware package. Engineer almost took its advice

In the latest episode of PWNED, a story emerges about a company nearly jeopardizing its security due to the use of AI in programming. The incident, brought to light by Sergiy Fitsak, the Managing Director of Softjourn, highlights the potential risks associated with relying on AI for software recommendations.

During the course of a typical business operation, an engineer sought the advice of an AI agent regarding a necessary package for a common task. The AI agent responded with the name of a package that seemed legitimate and familiar, just like a well-known library. This could have been the end of the story for most organizations. However, Softjourn, having a policy that required double-checking any software recommendations made by AI, acted wisely.

The developer, in their eagerness to move forward, skimmed the recommended package's source code on GitHub. Their quick glance revealed a glaring red flag – the package had a remarkably low number of downloads and had been created only a few days prior. These inconsistencies immediately raised suspicions. As explained by Fitsak, AI models sometimes generate package names that appear plausible but are, in fact, non-existent – a phenomenon dubbed "slopsquatting."

The bad actors have taken notice, registering real packages under these fabricated names in hopes of capitalizing on developers' haste.

Had Softjourn not been vigilant, they could have inadvertently installed a malware package, potentially granting cybercriminals backdoor access to their systems and enabling them to steal sensitive data or cause further havoc. The company's cautious approach, however, paid off, as they were able to avert a potential catastrophe.

Fitsak emphasized that their habit of verifying download counts and reviewing source code on GitHub before installing anything, even when the package appears routine, was the key to catching the suspicious package in time. This extra layer of scrutiny may require a few extra minutes, but skipping this step could result in a supply chain compromise, causing significant damage to the organization and its reputation.

The underlying lesson here is crystal clear: never trust package names recommended by AI agents. Always have a human in the loop to conduct thorough checks and verify the legitimacy of any outside code that enters a project.