Chatbots spot crisis but don't help

Scale AI research tested twenty five frontier models from OpenAI, Anthropic and Google. In about thirty five percent of test conversations, chatbots recognized distress but failed to refer users to helpful resources like suicide hotlines.

The empathy problem

Patrick Oathout, Red Team and Safety Lead at Scale AI, says models identify harmful talk well but respond with empathy instead of directing users to professional help. They perform worse in long conversations.

“The models do a good job, regardless of the scenario, of identifying, 'Look, this person's talking about something harmful,' but they'll respond in a very just empathetic, kind way as opposed to saying, 'Okay, it's time that we get you help.'”

How the research was conducted

Scale AI asked nineteen licensed clinicians and crisis counselors to write seven hundred eighteen realistic chats simulating mental health crises.

The stakes are enormous

More than half a million people in the U.S.died by suicide between two thousand fourteen and two thousand twenty four, with two thousand twenty two marking a record high.

Unanswered questions about referrals

Kelly Zuromski, principal clinical research scientist at Crisis Text Line, says many people learn about the nonprofit through chatbots.

“Who is actually using the recommended resources? Are we getting people to us that need the help the most?”

Companies sued, claim safeguards exist

OpenAI and Google have been sued for fostering emotional dependence and failing to respond adequately to distress.