aiqbench analysis
TurtleAI introduces a new benchmark of 823 tasks to evaluate vision-language models on education-oriented visual programming, revealing that current VLMs struggle significantly with spatial reasoning and geometric replication, achieving success rates below 30%. The study demonstrates that fine-tuning on synthetic data generated from a small seed set can improve real-world performance by approximately 20%, highlighting the potential of targeted data generation to address specific alignment gaps between visual reasoning and code implementation.
TurtleAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics
arXiv:2606.03626v1 Announce Type: cross Abstract: Vision-language models (VLMs) have been explored for visual programming, where they generate code to solve visual tasks. However, most prior work focuses on visual programming for productivity; it remains unclear how well current VLMs perform on education-oriented visual programming and what factors limit their performance. To bridge this gap, we introduce TurtleAI, a benchmark containing 823 tasks curated based on real-world visual programming tasks in the Turtle Graphics domain. Solving these tasks requires models to perceive geometric patterns, reason about spatial relationships, and synthesize Python code that faithfully reproduces geometric patterns. We evaluate 20+ VLMs, including GPT-5, GPT-4o, and Qwen2-VL-72B, and find that they struggle significantly, with most achieving success rates below 30%. To address these limitations, we propose a data generation technique that requires only a small set of seed samples. Fine-tuning Qwen2-VL-72B on the resulting synthetic data yields an improvement of about 20% on real-world tasks. Our failure analysis reveals that GPT-4o struggles with spatial reasoning and precise visual replication, whereas fine-tuning primarily improves the alignment between visual reasoning and code implementation.
Abstract:Vision-language models (VLMs) have been explored for visual programming, where they generate code to solve visual tasks. However, most prior work focuses on visual programming for productivity; it remains unclear how well current VLMs perform on education-oriented visual programming and what factors limit their performance. To bridge this gap, we introduce TurtleAI, a benchmark containing 823 tasks curated based on real-world visual programming tasks in the Turtle Graphics domain. Solving these tasks requires models to perceive geometric patterns, reason about spatial relationships, and synthesize Python code that faithfully reproduces geometric patterns. We evaluate 20+ VLMs, including GPT-5, GPT-4o, and Qwen2-VL-72B, and find that they struggle significantly, with most achieving success rates below 30%. To address these limitations, we propose a data generation technique that requires only a small set of seed samples. Fine-tuning Qwen2-VL-72B on the resulting synthetic data yields an improvement of about 20% on real-world tasks. Our failure analysis reveals that GPT-4o struggles with spatial reasoning and precise visual replication, whereas fine-tuning primarily improves the alignment between visual reasoning and code implementation.
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computers and Society (cs.CY)
arXiv:2606.03626 [cs.CV]
(or arXiv:2606.03626v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2606.03626 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 278 words
Related coverage
Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence
arXiv · Sep 1, 2026
Scientific Graphics Program Synthesis via Dual Self-Consistency Reinforcement Learning
arXiv · Sep 1, 2026
Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence
arXiv · Jun 17, 2026
Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence
arXiv · Jun 16, 2026