The Implications of Linguistic Illegibility for LLM Security

LLMs are trained to generate natural language. However, there are strong indications that an LLM's externalized linguistic outputs and mechanistically-extracted linguistic features may not reliably reflect the model's internal computations. The term "linguistic illegibility" is introduced to describe situations where an LLM's externalized or mechanistically-probed language artifacts do not accurately represent how the model processes information.

It is argued that for LLMs whose internal computations are not directly expressed through language, but rather through mathematical operations over activation spaces, linguistic illegibility is an unavoidable reality. This has significant implications for security mechanisms that rely on a model's linguistic self-reporting, such as chain-of-thought monitoring, constitutional self-critique, and activation probing for linguistically-defined feature vectors.

Since linguistic illegibility is always a possibility, these security mechanisms cannot be completely sound. The model sandbox will always require isolation techniques whose guarantees do not depend on reading the model's linguistic state. Observing a model's outputs using taint tracking is presented as a promising approach for an effective sandbox.

Taint tracking allows defining, a priori, various pieces of system state that should never be influenced by model-produced data, regardless of how the model linguistically self-reports. Additional sandboxing mechanisms, such as robust virtualization and third-party auditing of sandboxing configurations, are also discussed. These mechanisms, when combined, provide a critical floor beneath linguistic monitoring, potentially mitigating recent sandbox exploits by frontier models.