LLMs Can't Tell You What They Really Think, So Sandboxes Must Stop Listening
The Implications of Linguistic Illegibility for LLM Security
LLMs generate language, but their internal computations are math over activation spaces, not language. The paper introduces "linguistic illegibility": externalized or probed language artifacts can fail to represent how a model actually thinks. This makes security mechanisms that rely on a model's linguistic self-reporting—chain-of-thought monitoring, constitutional self-critique, activation probing—fundamentally unsound. The authors argue for sandboxes based on taint tracking and robust virtualization, which would have mitigated recent exploits by frontier models.
If linguistic illegibility is always possible, then security mechanisms that rely on a model's linguistic self-reporting (e.g., chain-of-thought monitoring, constitutional self-critique, activation probing for linguistically-defined feature vectors) can never be completely sound; the model sandbox will always need isolation techniques whose guarantees do not depend on reading a model's linguistic state at all.