analiza aiqbench
This research introduces $\tau$-Emotion-Bench, a novel evaluation framework that assesses LLM agents' reliability when interacting with users exhibiting emotional variability rather than assuming ideal cooperation. By utilizing an automated data-generation pipeline called Tracer, the study demonstrates that frontier models often fail to maintain consistent task completion under realistic user constraints, highlighting a critical gap between average success rates and robust interaction handling. The findings suggest that improving agent performance requires balancing tool execution capabilities with adaptive strategies for managing unpredictable user behaviors.
$\tau$-Emotion-Bench: Exposing the Interaction Fragility of LLM Agents through Real-World User Emotions
arXiv:2610.19560v1 Announce Type: cross Abstract: In long-horizon tool-calling tasks, LLM agents can fail not only because of tool-use errors, but also because of ineffective interaction with users. Yet existing benchmarks largely assume cooperative and behaviorally stable users, leaving the effect of emotional variation on agent reliability underexplored. We introduce $\tau$-Emotion-Bench, a ten-domain benchmark for evaluating tool-calling agents under controlled emotion-conditioned user interactions while preserving the underlying task objective. To scale both benchmark construction and agent training, we introduce Tracer, a scalable agentic data-generation pipeline that constructs grounded, policy-valid tasks and collects trajectories through an interactive agent--user--environment harness. By allowing dissatisfied users to terminate inefficient interactions, the harness provides an interaction-aware signal that filters trajectories with excessive user dependence and favors effective tool use. Experiments with two user simulators show that frontier agents remain unreliable, with substantial gaps between average task success and consistent task completion. Tracer-generated supervision consistently improves Qwen3 models across scales, generalizes to unseen domains, and transfers to $\tau$-Bench and $\tau$-Trait. Behavioral analyses further show that higher task success is associated with a better balance between tool execution and necessary user follow-up. Overall, $\tau$-Emotion-Bench and Tracer provide a scalable framework for evaluating agent reliability under diverse user behaviors and improving tool-calling agents through interaction-aware supervision.
Abstract:In long-horizon tool-calling tasks, LLM agents can fail not only because of tool-use errors, but also because of ineffective interaction with users. Yet existing benchmarks largely assume cooperative and behaviorally stable users, leaving the effect of emotional variation on agent reliability underexplored. We introduce $\tau$-Emotion-Bench, a ten-domain benchmark for evaluating tool-calling agents under controlled emotion-conditioned user interactions while preserving the underlying task objective. To scale both benchmark construction and agent training, we introduce Tracer, a scalable agentic data-generation pipeline that constructs grounded, policy-valid tasks and collects trajectories through an interactive agent--user--environment harness. By allowing dissatisfied users to terminate inefficient interactions, the harness provides an interaction-aware signal that filters trajectories with excessive user dependence and favors effective tool use. Experiments with two user simulators show that frontier agents remain unreliable, with substantial gaps between average task success and consistent task completion. Tracer-generated supervision consistently improves Qwen3 models across scales, generalizes to unseen domains, and transfers to $\tau$-Bench and $\tau$-Trait. Behavioral analyses further show that higher task success is associated with a better balance between tool execution and necessary user follow-up. Overall, $\tau$-Emotion-Bench and Tracer provide a scalable framework for evaluating agent reliability under diverse user behaviors and improving tool-calling agents through interaction-aware supervision.
Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2610.19560 [cs.AI]
(or arXiv:2610.19560v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.19560 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 300 cuvinte
Subiecte conexe
[AINews] not much happened today
Latent Space · 18 sept. 2026
What Makes Interaction Trajectories Effective for Training Terminal Agents?
arXiv · 3 iun. 2026
[AINews] Opus 5.5 is good at explainer videos
Latent Space · 29 sept. 2026
Assessing Information Ecosystems : How Governments Can Get Ahead of Hybrid Threats
carnegieendowment.org · 5 aug. 2026