analiza aiqbench
This study reveals that production LLM agents currently rely on simple, controllable architectures, with most executing under ten steps and avoiding weight tuning in favor of prompting off-the-shelf models. Practitioners prioritize reliability as the primary challenge, addressing it through systems-level design rather than model optimization. These findings provide the research community with a realistic baseline for agent deployment, highlighting a significant gap between current practices and the need for more robust evaluation frameworks.
Measuring Agents in Production
arXiv:2512.04123v4 Announce Type: replace-cross Abstract: LLM-based agents already operate in production across many industries, yet we lack an understanding of what technical methods make deployments successful. We present the first systematic study of Measuring Agents in Production, MAP, using first-hand data from agent developers. We conducted 20 case studies via in-depth interviews and surveyed 86 deployed systems practitioners across 26 domains. We investigate why organizations build agents, how they build them, how they evaluate them, and their top development challenges. Our study finds that production agents are built using simple, controllable approaches: 68% execute at most 10 steps before human intervention, 70% rely on prompting off-the-shelf models instead of weight tuning, and 74% depend primarily on human evaluation. Reliability (consistent correct behavior over time) remains the top development challenge, which practitioners currently address through systems-level design. MAP documents the current state of production agents, providing the research community with visibility into deployment realities and underexplored research avenues.
Authors:Melissa Z. Pan, Negar Arabzadeh, Riccardo Cogo, Yuxuan Zhu, Alexander Xiong, Lakshya A Agrawal, Huanzhi Mao, Emma Shen, Sid Pallerla, Liana Patel, Shu Liu, Tianneng Shi, Xiaoyuan Liu, Jared Quincy Davis, Emmanuele Lacavalla, Alessandro Basile, Shuyi Yang, Paul Castro, Daniel Kang, Koushik Sen, Dawn Song, Joseph E. Gonzalez, Ion Stoica, Matei Zaharia, Marquita Ellis
Abstract:LLM-based agents already operate in production across many industries, yet we lack an understanding of what technical methods make deployments successful. We present the first systematic study of Measuring Agents in Production, MAP, using first-hand data from agent developers. We conducted 20 case studies via in-depth interviews and surveyed 86 deployed systems practitioners across 26 domains. We investigate why organizations build agents, how they build them, how they evaluate them, and their top development challenges. Our study finds that production agents are built using simple, controllable approaches: 68% execute at most 10 steps before human intervention, 70% rely on prompting off-the-shelf models instead of weight tuning, and 74% depend primarily on human evaluation. Reliability (consistent correct behavior over time) remains the top development challenge, which practitioners currently address through systems-level design. MAP documents the current state of production agents, providing the research community with visibility into deployment realities and underexplored research avenues.
Computers and Society (cs.CY); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Software Engineering (cs.SE)
arXiv:2512.04123 [cs.CY]
(or arXiv:2512.04123v4 [cs.CY] for this version)
Text integral sub licența deschisă a publicației · 227 cuvinte
Subiecte conexe
Amazon Research Awards recipients announced
Amazon Science · 27 mai 2026
34 Amazon Research Awards Build on Trainium recipients announced
Amazon Science · 5 aug. 2026
Amazon is investing in the Lean Focused Research Organization
Amazon Science · 26 iul. 2026
Researchers Find Nonlocal Measurement Without Qubit Entanglement
Quantum Zeitgeist · 2 oct. 2026