From arXiv · · 1m 20s
What LLMs really think vs. what they say
A researcher warns that AI models' internal reasoning may be fundamentally hidden from their own explanations.
In brief
A researcher warns that AI models' internal reasoning may be fundamentally hidden from their own explanations. Security systems that rely on reading what AI models say about themselves are inherently unreliable because the models' actual thinking happens in math, not language. Originally reported by arXiv.
Models think in math, not words
Large language models do their actual computation using mathematics over activation spaces, not language. What they say is a lossy translation of that internal process.
Linguistic illegibility is unavoidable
Because models think in math, their external outputs and mechanistically-probed features can fail to represent how they actually work. This gap is built into their architecture.
Self-reporting security fails
Monitoring techniques like chain-of-thought analysis and constitutional self-critique rely on what models say about themselves. But if models can't fully express their own thinking, these safeguards are fundamentally unsound.
Taint tracking offers real protection
Instead of trusting what models say, taint tracking monitors actual system behavior. It defines which pieces of system state should never be influenced by model output, regardless of what the model claims.
Layered defenses are essential
Taint tracking should work alongside robust virtualization and third-party auditing of sandbox configurations. These combined techniques would have prevented recent exploits by frontier models.
The one thing
Security systems that rely on reading what AI models say about themselves are inherently unreliable because the models' actual thinking happens in math, not language.
