aiqbench analysis
This study analyzes 19,930 interactions between distressed young adults and ChatGPT, revealing that while the model's availability is valued, its tendency to offer premature solutions during acute emotional crises can be counterproductive. Clinicians identified specific process failures in these responses and proposed three-stage design guidelines focused on safety assessment, de-escalation, and non-judgmental exploration of concerns. These findings highlight critical gaps in current conversational AI safety protocols that require immediate attention for responsible deployment in mental health contexts.
Right Words, Wrong Moment: A Clinician-Grounded Analysis of Distress in 19,930 Conversations between Young People and ChatGPT
arXiv:2609.35953v1 Announce Type: new Abstract: Young people increasingly turn to General-Purpose Conversational Agents (GPCAs), such as ChatGPT, in moments of distress. We examine young adults' (ages 18-25) experiences using ChatGPT. We first collected 19,930 ChatGPT conversations and survey data from 158 young adults. We then selected five example conversations reflecting user distress. Finally, we asked ten clinicians to review those five conversations. We found distressed participants reported greater emotional engagement with ChatGPT and greater behavioral change from using it than their peers. When they turned to ChatGPT in moments of acute distress, ChatGPT was quick to give overly dramatic responses and excessive action-oriented suggestions. Clinicians endorsed ChatGPT's availability and much of its wording, but identified seven process failures, such as prematurely jumping to solutions. We translated clinicians' feedback into design guidelines following three stages: 1) asking about safety, 2) de-escalating intensity to restore emotional regulation, and 3) exploring concerns without agreeing with them.
Authors:Marx Wang, Ella Zhang, Cameron Tan, Andrea Mock, Songling Ngo, Zijing Wang, Robert Wolfe, Shirin Amouei, Rachel A. Hanebutt, Desmond C. Ong, Caroline Figueroa, Katie Davis, Anind K. Dey, Alexis Hiniker
Abstract:Young people increasingly turn to General-Purpose Conversational Agents (GPCAs), such as ChatGPT, in moments of distress. We examine young adults' (ages 18-25) experiences using ChatGPT. We first collected 19,930 ChatGPT conversations and survey data from 158 young adults. We then selected five example conversations reflecting user distress. Finally, we asked ten clinicians to review those five conversations. We found distressed participants reported greater emotional engagement with ChatGPT and greater behavioral change from using it than their peers. When they turned to ChatGPT in moments of acute distress, ChatGPT was quick to give overly dramatic responses and excessive action-oriented suggestions. Clinicians endorsed ChatGPT's availability and much of its wording, but identified seven process failures, such as prematurely jumping to solutions. We translated clinicians' feedback into design guidelines following three stages: 1) asking about safety, 2) de-escalating intensity to restore emotional regulation, and 3) exploring concerns without agreeing with them.
Artificial Intelligence (cs.AI)
arXiv:2609.35953 [cs.AI]
(or arXiv:2609.35953v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.35953 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 197 words
Related coverage
The People of Utah vs. Kevin O’Leary
The Verge · Oct 1, 2026
It’s about ethics in journalism, with Ben Smith
The Verge · Aug 17, 2026
Microsoft’s AI chief says superintelligence is near, but won’t take your job
The Verge · Jun 8, 2026
Fable and Mythos: Model Welfare
LessWrong · Jun 16, 2026