From Out of the Software Crisis · · 1m 40s
The LLMentalist Effect: How Chatbots Mimic Psychic Con Tricks
Chatbots trick users into thinking they’re intelligent using statistical guesses.
In brief
Chatbots trick users into thinking they’re intelligent using statistical guesses. Chatbots create false intelligence by mimicking psychic con tactics, not actual reasoning. Originally reported by Out of the Software Crisis.
The Intelligence Illusion
Many believe chatbots are intelligent, but LLMs lack reasoning mechanisms.
LLMs Are Math Models
LLMs predict text responses using statistical patterns, not real thought.
The Psychic’s Con
Psychic scams use validation statements to trick people into believing they read minds.
Validation Statements
Statements like 'you tend to be hard on yourself' sound specific but are statistically generic.
The LLMentalist Effect
Chatbots mimic psychic con tactics to create false intelligence in users.
Audience Selection
Tech enthusiasts and AGI believers are more likely to believe chatbots are intelligent.
Setting the Stage
AI hype primes users to expect intelligence in chatbots, ignoring warnings about hallucinations.
Prompt Interaction
Initial chatbot responses are statistically plausible but not truly specific to users.
Subjective Validation
Users interpret generic responses as specific to them, creating false intelligence.
Reinforcement Learning
RLHF training rewards responses that sound accurate, not factually correct.
The Validation Loop
Users keep validating chatbot responses, reinforcing false intelligence beliefs.
Mark Becomes Cheerleader
Believers in AGI become convinced chatbots have true intelligence.
Intelligence Boosts the Effect
Smarter users are more likely to fall for the illusion by rationalizing it.
The Illusion is Accidental
AI researchers unintentionally created the psychic con effect through RLHF.
Pseudoscience in AI
Tech companies apply psychic scams to health, hiring, and predictions.
The Real Risk
Using chatbots for decisions risks creating false intelligence beliefs.
The short version
Chatbots create false intelligence by mimicking psychic con tactics, not actual reasoning.
