analiza aiqbench
Despite recent strides in code generation, current frontier LLM agents struggle to reliably automate complex Office tasks, achieving only 68.8% of the performance benchmark set by human professionals. The study reveals that even advanced agentic systems with iterative repair mechanisms fall significantly short of the 95.5% score required for practical proficiency in long-horizon planning and multi-application integration. This gap highlights that fine-grained document automation remains a critical unsolved challenge for deploying LLMs in professional productivity environments.
Mind the Gap: Can Frontier LLMs Pass a Standardized Office Proficiency Exam?
arXiv:2606.10956v1 Announce Type: cross Abstract: The deployment of Large Language Model (LLM) agents for computer automation is accelerating, yet their ability to navigate complex, professional-grade productivity software is largely untested. We argue that Office automation is an ideal environment for benchmarking document-automation capability, as it requires long-horizon planning and reasoning, precise parameter configuration, and multi-application integration. To quantify this capability, we introduce an evaluation based on China's National Computer Rank Examination (NCRE), featuring 200 comprehensive practical-operation tasks across Word, Excel, and PowerPoint. Each task is scored on a 100-point rubric scale using 7,118 machine-gradable criteria, and Score Rate (SR) denotes the mean percentage of rubric points earned across these tasks. We benchmark 7 frontier LLMs and observe stark limitations: single-turn models score a maximum of 36.6%. A stronger agentic system with execution feedback, iterative repair, and broader Office automation access reaches 68.8%, but remains below the 95.5% community-reference score used as a scoring sanity check. Ultimately, our experiments demonstrate that despite recent advancements in code generation, achieving reliable fine-grained Office document automation remains a significant challenge for current code-generating LLM and agent systems.
Abstract:The deployment of Large Language Model (LLM) agents for computer automation is accelerating, yet their ability to navigate complex, professional-grade productivity software is largely untested. We argue that Office automation is an ideal environment for benchmarking document-automation capability, as it requires long-horizon planning and reasoning, precise parameter configuration, and multi-application integration. To quantify this capability, we introduce an evaluation based on China's National Computer Rank Examination (NCRE), featuring 200 comprehensive practical-operation tasks across Word, Excel, and PowerPoint. Each task is scored on a 100-point rubric scale using 7,118 machine-gradable criteria, and Score Rate (SR) denotes the mean percentage of rubric points earned across these tasks. We benchmark 7 frontier LLMs and observe stark limitations: single-turn models score a maximum of 36.6%. A stronger agentic system with execution feedback, iterative repair, and broader Office automation access reaches 68.8%, but remains below the 95.5% community-reference score used as a scoring sanity check. Ultimately, our experiments demonstrate that despite recent advancements in code generation, achieving reliable fine-grained Office document automation remains a significant challenge for current code-generating LLM and agent systems.
Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2606.10956 [cs.AI]
(or arXiv:2606.10956v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2606.10956 arXiv-issued DOI via DataCite (pending registration)
Text integral sub licența deschisă a publicației · 200 cuvinte
Subiecte conexe
Research Note: Filtering Subversion-Relevant Information From Pretraining Data Is Feasible
LessWrong · 5 oct. 2026
The Case Against Slowing Down AI
LessWrong · 29 sept. 2026
Reviewing failure modes in science as we prepare for AI researchers
LessWrong · 8 oct. 2026
If This Were a Test, How Much Would It Cost?
LessWrong · 16 iun. 2026