analiza aiqbench
This article is squarely about AI: an evaluation benchmark (Mr Dre) for Deep Research Agents, which are LLM-based systems, testing their ability to revise long-form reports across multiple turns. The core finding is that agents fix most requested feedback but regress on 16-28% of previously covered content and citation quality, and that prompt engineering and a dedicated revision subagent do not fully resolve this. For readers, the takeaway is that multi-turn reliability, not just single-shot writing quality, should be a standard metric when judging research agents before deploying them in real workflows.
Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revision
arXiv:2601.13217v2 Announce Type: replace-cross Abstract: Existing benchmarks for Deep Research Agents (DRAs) treat report generation as a single-shot writing task, which fundamentally diverges from how human researchers iteratively draft and revise reports via self-reflection or peer feedback. Whether DRAs can reliably revise reports with user feedback remains unexplored. We introduce Mr Dre, an evaluation suite that establishes multi-turn report revision as a new evaluation axis for DRAs. Mr Dre consists of (1) a unified long-form report evaluation protocol spanning comprehensiveness, factuality, and presentation, and (2) a human-verified feedback simulation pipeline for multi-turn revision. Our analysis of five diverse DRAs reveals a critical limitation: while agents can address most user feedback, they also regress on 16-28% of previously covered content and citation quality. Over multiple revision turns, even the best-performing agents leave significant headroom, as they continue to disrupt content outside the feedback's scope and fail to preserve earlier edits. We further show that these issues are not easily resolvable through inference-time fixes such as prompt engineering and a dedicated subagent for report revision.
Abstract:Existing benchmarks for Deep Research Agents (DRAs) treat report generation as a single-shot writing task, which fundamentally diverges from how human researchers iteratively draft and revise reports via self-reflection or peer feedback. Whether DRAs can reliably revise reports with user feedback remains unexplored. We introduce Mr Dre, an evaluation suite that establishes multi-turn report revision as a new evaluation axis for DRAs. Mr Dre consists of (1) a unified long-form report evaluation protocol spanning comprehensiveness, factuality, and presentation, and (2) a human-verified feedback simulation pipeline for multi-turn revision. Our analysis of five diverse DRAs reveals a critical limitation: while agents can address most user feedback, they also regress on 16-28% of previously covered content and citation quality. Over multiple revision turns, even the best-performing agents leave significant headroom, as they continue to disrupt content outside the feedback's scope and fail to preserve earlier edits. We further show that these issues are not easily resolvable through inference-time fixes such as prompt engineering and a dedicated subagent for report revision.
ACL 2026 (SAC Highlight Award). 43 pages. Code and data: this https URL
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2601.13217 [cs.CL]
(or arXiv:2601.13217v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2601.13217 Focus to learn more arXiv-issued DOI via DataCite
https://doi.org/10.18653/v1/2026.acl-long.609 Focus to learn more DOI(s) linking to related resources
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 287 cuvinte