How Cross-Model Compatibility Lets Attackers Extract Proprietary LLM Reasoning Traces

The research paper "How Cross-Model Compatibility Lets Attackers Extract Proprietary LLM Reasoning Traces" reveals a concerning flaw in major AI companies' attempts to safeguard proprietary reasoning from LLMs. Major players like OpenAI, Anthropic, and Google now offer users a "step-by-step reasoning" feature, which shows why a model reached a conclusion. However, this reasoning also poses risks, as it is valuable to competitors, researchers, and attackers who wish to understand how these models think.

To address this, companies encrypt reasoning traces on the server before sending them to the user's device. The encrypted blocks, called "reasoning tokens," travel between the user's device and the server when the user makes follow-up requests. While this appears secure, researchers discovered that the encryption does not truly hide the reasoning. Instead, it merely makes it appear hidden, as encrypted blocks designed to work across different sessions and models for convenience create an architectural vulnerability.

The main issue lies in the universal compatibility of these encrypted tokens. Tokens from different models, such as GPT-4 and GPT-3.5, or Claude Instant and Claude 3, are interchangeable. This interoperability simplifies the system but exposes a fatal flaw. An attacker can exploit this by obtaining an encrypted reasoning token from a stronger model (e.g., GPT-4) and injecting it into a weaker model (e.g., GPT-3.5).

When the weaker model processes the token, it decrypts it and outputs the reasoning in plaintext, bypassing the stronger model's safeguards.

The researchers demonstrated this attack across all three companies, showing that reasoning from frontier models can be successfully extracted through this method. The attack requires only API access to a weaker model, which is easily obtainable. The attacker does not need a breach, internal access, or detailed knowledge of the encryption scheme.

All they need is a token from the stronger model. This vulnerability is particularly dangerous as the attacker can obtain encrypted reasoning tokens from various sources, including development environments, error traces, online examples, and public repositories.

In summary, the cross-model compatibility of encrypted reasoning tokens creates a significant security risk, enabling attackers to extract proprietary LLM reasoning traces. The seemingly secure architecture, designed for convenience, has inadvertently opened the door for such attacks, putting the proprietary algorithms and reasoning patterns of these AI models at risk.