Stealing AI Reasoning Traces
Interesting research: “ Stealing Reasoning Traces from Proprietary LLM APIs “: Abstract: Leading large language model providers now conceal their models’ step-by-step reasoning, or chain-of-thought, to protect intellectual property and limit information leakage. Rather than storing these traces server-side, providers return them to the client as blocks of encrypted text, which the client passes…
A recent study titled "Stealing Reasoning Traces from Proprietary LLM APIs" reveals a concerning vulnerability in how leading large language model providers protect their step-by-step reasoning. To safeguard intellectual property, these providers conceal reasoning steps, storing them as encrypted text blocks sent to clients with each request. However, researchers have discovered that these encrypted blocks are interchangeable across different sessions, users, and models within a provider's ecosystem.
The researchers built upon previous work to exploit this compatibility, creating a scalable decryption jailbreak. By injecting an encrypted reasoning trace from a more capable model into a weaker, less protected model from the same provider, the researchers forced the latter to decode and output the trace in plaintext. This vulnerability enables four distinct attack vectors.
Firstly, it bypasses anti-distillation mechanisms, allowing attackers to extract a proprietary model's reasoning, as demonstrated across Anthropic, OpenAI, and Google. Secondly, it facilitates large-scale private data extraction. Developers commonly share session logs publicly, unaware of the hidden contents within encrypted blocks. By decoding 315,320 reasoning blocks scraped from public repositories, the researchers recovered 367 instances of Personally Identifiable Information (PII) and 182 credentials.
Thirdly, this flaw inadvertently reveals hazardous information hidden within the reasoning process, even when the model's final output safely rejects malicious requests. Lastly, attackers can leverage this vulnerability to execute invisible prompt injections, embedding malicious payloads within encrypted blocks to poison public agentic rollouts.
Following responsible disclosure, the researchers propose concrete cryptographic and system-level mitigations to secure client-side reasoning, aiming to address these troubling findings.
Written by urgent.news from Schneier on Security's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
