analiza aiqbench
This benchmark introduces PPTBench, a rigorous testbed for visual coding agents that evaluates their ability to reconstruct editable scientific diagrams from arXiv papers. The study reveals a significant gap between generating syntactically valid slide files and achieving high semantic or visual fidelity, particularly in rendering text details. These findings highlight the current limitations of AI models when tasked with precise, structured reconstruction of complex visual knowledge artifacts.
PPTBench: Can Coding Agents Reconstruct the Visual World through Structured, Editable Slides
arXiv:2609.29718v1 Announce Type: new Abstract: Coding agents are beginning to act in the visual world. They now build webpages, GUIs, games, 3D scenes, diagrams, and documents. Success in such visual coding requires bridging two spaces: inferring visual structure and expressing it programmatically. Slides are a core medium of knowledge work, widely used to communicate ideas and collaborate in a form that people can directly inspect and edit. Therefore, they provide an ideal testbed for visual coding, as they require agents to recover visual structure and realize it as editable objects. However, existing benchmarks either rely on subjective open-ended evaluation, produce non-editable code outputs, or focus only on local editing rather than end-to-end visual reconstruction. We introduce PPTBench, which benchmarks visual coding through editable slide reconstruction. It contains 500 tasks, each based on a scientific flow diagram from a real arXiv paper and requiring agents to reconstruct it as a single PPTX page composed of native, editable objects. A four-stage Agentic Judge evaluates artifact validity, semantic correctness, rendering quality, and fine-grained visual quality. Across 31 configurations spanning model families, effort levels, and harnesses, the best configuration, Kimi K3, reaches only 67.80, while the median scores 19.47. We find that agents can reliably produce valid PPTX files but still struggle with semantic and visual correctness, especially text details. More reasoning mainly helps agents pass hard gates, while stronger verification is more consistently associated with higher quality. PPTBench advances the vision of coding agents that can understand and reconstruct the visual world through structured, editable code.
Abstract:Coding agents are beginning to act in the visual world. They now build webpages, GUIs, games, 3D scenes, diagrams, and documents. Success in such visual coding requires bridging two spaces: inferring visual structure and expressing it programmatically. Slides are a core medium of knowledge work, widely used to communicate ideas and collaborate in a form that people can directly inspect and edit. Therefore, they provide an ideal testbed for visual coding, as they require agents to recover visual structure and realize it as editable objects. However, existing benchmarks either rely on subjective open-ended evaluation, produce non-editable code outputs, or focus only on local editing rather than end-to-end visual reconstruction. We introduce PPTBench, which benchmarks visual coding through editable slide reconstruction. It contains 500 tasks, each based on a scientific flow diagram from a real arXiv paper and requiring agents to reconstruct it as a single PPTX page composed of native, editable objects. A four-stage Agentic Judge evaluates artifact validity, semantic correctness, rendering quality, and fine-grained visual quality. Across 31 configurations spanning model families, effort levels, and harnesses, the best configuration, Kimi K3, reaches only 67.80, while the median scores 19.47. We find that agents can reliably produce valid PPTX files but still struggle with semantic and visual correctness, especially text details. More reasoning mainly helps agents pass hard gates, while stronger verification is more consistently associated with higher quality. PPTBench advances the vision of coding agents that can understand and reconstruct the visual world through structured, editable code.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2609.29718 [cs.CL]
(or arXiv:2609.29718v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.29718 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 346 cuvinte
