Can LLM Agents Resolve Service Failures? Simulating Multi-Turn Complaint Handling under Information Asymmetry

Title:Can LLM Agents Resolve Service Failures? Simulating Multi-Turn Complaint Handling under Information Asymmetry

Abstract:LLM agents are widely adopted in service chatbots for rule-guided support and information access, yet emotionally demanding complaint handling still relies heavily on human agents. We introduce ServiceBench, to our knowledge the first benchmark dedicated to complaint handling, with 2,500 scenarios spanning 184 complaint categories and five service industries. We propose Asymmetric Progressive Complaint Simulation (APCS) to simulate realistic complaint handling, together with a justice-based evaluation framework. Our experiments reveal that even capable LLM agents struggle to handle complaints effectively. They lack the ability to proactively gather information and strategically guide conversations toward resolution, often resorting to premature compensation. We further show that targeted post-training with SFT and DPO substantially improves these capabilities, enabling a smaller model to match or surpass much larger models on several outcomes.

Artificial Intelligence (cs.AI)

arXiv:2610.24733 [cs.AI]

(or arXiv:2610.24733v1 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2610.24733 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.