LiteLLM supply chain attack started with an exposed master key

Last week's supply chain attack against LiteLLM began with a master key that was inadvertently pushed to a public GitHub repository in May. The attack escalated when an individual uploaded a tainted version (1.107.0) of the well-known AI gateway to PyPI and its container registries on Wednesday. This malicious version included an information-stealing trojan and a remote access trojan that could execute arbitrary code on compromised machines.

Although the infected package was quarantined within two hours of its release, it amassed around 49,000 downloads. The post-mortem investigation traced the attack chain back to May 27 when the sensitive master key was accidentally committed to the public LiteLLM-Labs/lite-harness repository. The developers quickly identified the mistake and removed the exposed commit, but neglected to revoke the master key.

Subsequently, the threat actor gained ownership of all repositories on LiteLLM's GitHub platform. This situation is particularly concerning because GitHub CLI tokens, which are "post-2fa," do not activate GitHub's two-factor authentication required for high-impact repository actions. Consequently, the attacker managed to insert malicious commits into branches in other internal repositories, including workflows aimed at stealing any tokens and secrets used during the continuous integration development process.

The attacker obtained additional GitHub CLI OAuth tokens associated with two engineers who had maintainer privileges and accessed several other internal systems. By examining the actor's activities, LiteLLM found attempts to identify and exfiltrate further keys and tokens, primarily focusing on upstream inference API tokens. The miscreant, having amassed several privileged identities, was able to make changes to GitHub repositories without raising suspicion due to the actions and approvals appearing to come from legitimate engineers.

In response to the incident, LiteLLM is revoking all tokens connected to internal systems and vendor platforms. The developers emphasized the importance of rotating potentially exposed keys, stating that this incident demonstrates the risks of failing to do so. The malicious package contained various infostealers capable of acquiring sensitive information like browser credentials and cookies from popular browsers, as well as AWS credentials, Google Cloud Platform service account keys, and Microsoft Azure tokens.

Additionally, it stole API keys for OpenAI, Anthropic, Cohere, and other large language model providers. If you installed and ran version 1.107.0, you should assume that any credentials accessible to the affected system might have been stolen. It is recommended to investigate affected systems, change passwords, and rotate exposed keys.