aiqbench analysis
PieArena introduces a robust benchmark for evaluating language agents in negotiation by incorporating behavioral profiles that assess instruction compliance, accuracy, and reputation beyond simple outcome metrics. The study reveals that while frontier models match human baselines, significant performance gains from agentic scaffolding are concentrated in mid- and lower-tier models, suggesting diminishing returns for top-tier systems in this specific domain. This work underscores the necessity of multi-dimensional evaluation frameworks to accurately gauge the strategic reasoning and economic value creation capabilities of AI agents in realistic business scenarios.
PieArena: Ranking and Profiling Language Agents in Realistic Negotiation Scenarios
arXiv:2602.05302v3 Announce Type: replace Abstract: We present an in-depth evaluation of LLMs' ability to negotiate, a central business task requiring strategic reasoning, theory of mind, and economic value creation. To do so, we introduce PieArena, a large-scale negotiation benchmark grounded in multi-agent interactions over realistic scenarios adapted from MBA negotiation courses at an elite business school. We evaluate language agents across three pairing regimes: mirror-play, cross-play, and human-LM play. We develop a ranking model for continuous negotiation payoffs that yields order-invariant, uncertainty-quantified leaderboards while correcting for systematic experimental asymmetries. We further study the effects of joint-intentionality agentic scaffolding and find asymmetric gains, with large improvements for mid- and lower-tier LMs and diminishing returns for frontier LMs. As calibration anchors, we collect human-human and human-LM negotiation data from trained business school students, finding that a representative frontier language agent (GPT-5) matches or exceeds this human baseline in our evaluation settings. Beyond deal outcomes, PieArena provides a multi-dimensional behavioral profile that reveals cross-model heterogeneity in instruction compliance, computation accuracy, as well as judge-assessed deception and reputation, illustrating the value of evaluation beyond outcome-only leaderboards.
Abstract:We present an in-depth evaluation of LLMs' ability to negotiate, a central business task requiring strategic reasoning, theory of mind, and economic value creation. To do so, we introduce PieArena, a large-scale negotiation benchmark grounded in multi-agent interactions over realistic scenarios adapted from MBA negotiation courses at an elite business school. We evaluate language agents across three pairing regimes: mirror-play, cross-play, and human-LM play. We develop a ranking model for continuous negotiation payoffs that yields order-invariant, uncertainty-quantified leaderboards while correcting for systematic experimental asymmetries. We further study the effects of joint-intentionality agentic scaffolding and find asymmetric gains, with large improvements for mid- and lower-tier LMs and diminishing returns for frontier LMs. As calibration anchors, we collect human-human and human-LM negotiation data from trained business school students, finding that a representative frontier language agent (GPT-5) matches or exceeds this human baseline in our evaluation settings. Beyond deal outcomes, PieArena provides a multi-dimensional behavioral profile that reveals cross-model heterogeneity in instruction compliance, computation accuracy, as well as judge-assessed deception and reputation, illustrating the value of evaluation beyond outcome-only leaderboards.
Artificial Intelligence (cs.AI)
arXiv:2602.05302 [cs.AI]
(or arXiv:2602.05302v3 [cs.AI] for this version)
Full text under the publisher's open licence · 188 words
Related coverage
Reputation, Strategy, and Emotion Effects on Generative AI Cooperation: A Comparison Across Reasoning and Non-Reasoning Models
arXiv · Oct 2, 2026
How LendingTree built a multi-agent mortgage assistant on Amazon Bedrock
AWS · Aug 5, 2026
PEMAND: Persona-Enriched Multi-Agent Negotiation for Household Decision-Making
arXiv · Aug 3, 2026
Inside the fight over Claude Mythos 5
The Verge · Jun 16, 2026