aiqbench analysis
This research addresses the critical bottleneck of form filling in pure-image domains where traditional OCR fails, demonstrating that current Vision-Language Agents struggle significantly with field localization. The authors introduce FieldFinder, a specialized tool that substantially improves agent accuracy by assisting LLMs in correctly identifying text placement on complex documents without relying on underlying DOM structures or typed PDF data. This work highlights the remaining gap between theoretical multi-modal capabilities and practical robustness for real-world document processing tasks.
FormGym: Doing Paperwork with Agents
arXiv:2506.14079v4 Announce Type: replace Abstract: Completing paperwork is a challenging and time-consuming problem. Form filling is especially challenging in the pure-image domain without access to OCR, typeset PDF text, or a DOM. For computer agents, it requires multiple abilities, including multi-modal understanding, information retrieval, and tool-use. We present a novel form-filling benchmark consisting of 432 fields spread across 55 documents and 3 tasks, requiring knowledge of 236 features per user. We find that baseline VLAs achieve less than 1% accuracy in most cases, primarily due to poor localization ability. GUI agents also struggle, scoring between 10.6-68.0% despite high cost and latency. Therefore, we also contribute FieldFinder, a tool to assist LLMs in identifying where to place text on a form. With FieldFinder, all models achieve equal or better performance in all six study conditions, with a maximum increase from 2% to 56%.
Abstract:Completing paperwork is a challenging and time-consuming problem. Form filling is especially challenging in the pure-image domain without access to OCR, typeset PDF text, or a DOM. For computer agents, it requires multiple abilities, including multi-modal understanding, information retrieval, and tool-use. We present a novel form-filling benchmark consisting of 432 fields spread across 55 documents and 3 tasks, requiring knowledge of 236 features per user. We find that baseline VLAs achieve less than 1% accuracy in most cases, primarily due to poor localization ability. GUI agents also struggle, scoring between 10.6-68.0% despite high cost and latency. Therefore, we also contribute FieldFinder, a tool to assist LLMs in identifying where to place text on a form. With FieldFinder, all models achieve equal or better performance in all six study conditions, with a maximum increase from 2% to 56%.
Artificial Intelligence (cs.AI)
arXiv:2506.14079 [cs.AI]
(or arXiv:2506.14079v4 [cs.AI] for this version)
Full text under the publisher's open licence · 148 words
Related coverage
Whack-a-drone
The Verge · Jul 24, 2026
5 Real-World Use Cases for AI Agents Transforming Industries
KDnuggets · Aug 21, 2026
I don't think Claude is misaligned in 'Agentic Misalignment Summer 2026 - Motivated Mislabeling'
LessWrong · Jul 17, 2026
Overcoming the Regulatory Bottleneck via Agent-to-Agent Protocols: A Nuclear Case Study
arXiv · Jun 9, 2026