Hundreds of AI apps were part of a network hiding sexually abusive deepfake tools

Apple has removed 19 applications that concealed dangerous AI tools behind common features. An investigation revealed a network of over 400 related apps and websites promising benign AI services such as pet portraits, language lessons, and fitness coaching. However, some of these apps covertly offered tools for generating sexually explicit images and videos from photos of real people.

This sophisticated setup used remote servers to determine which features individual users would see. Researchers found that apps within the network communicated with these servers after installation, transmitting data like the user's language, referral source, and a unique identifier. Depending on the user's route, they might see only innocuous features or be directed to a hidden website with explicit deepfake generators.

One trigger could be downloading an app after clicking a Facebook or Instagram ad. The network extended beyond Apple's platform, with 27 related apps on Google Play accumulating over 350,000 downloads. Many were rated for all ages, indicating the network's global reach. Investigators traced the operation back to Chinese-language developers and a Hong Kong-registered company called KeLing Technology Limited, though the exact control structure remained unclear.

Apple removed the 19 offending apps after being alerted and provided details of the network. However, the company emphasized that developers could alter the content supplied from external servers without needing to resubmit the app for review, allowing prohibited features to reappear after initial approval. In response to this revelation, Apple updated its guidelines on October 16, reinforcing restrictions against nonconsensual intimate imagery and hidden functionality.

These new rules explicitly address AI-generated nonconsensual intimate imagery, requiring developers to prevent or remove such content. Despite these changes, Apple still grapples with the challenge of detecting alterations made to hosted content after apps have been approved.