aiqbench analysis
This is an AI evaluation paper that scores complete agent systems on CAD tasks, including building parts from text and reconstructing them from engineering drawings. Its key methodological point is that the benchmark rewards geometric accuracy and specification consistency through a continuous score, not just whether the generated script runs. The results describe full model-agent pipelines rather than isolated models, so readers should treat the 61% leader score as a snapshot of one tooling setup, not a general measure of model capability.
Parametric CAD Bench v3: Evaluating Agentic Parametric CAD Authoring from Text and Engineering Drawings
arXiv:2611.02659v1 Announce Type: new Abstract: Producing an executable CAD script is not sufficient to establish that an agent has satisfied an engineering design request. Evaluation must also consider the resulting geometry, specified dimensions, and native CAD modeling history. We present Parametric CAD Bench v3, a 100-task benchmark comprising 30 text-to-CAD creation tasks, 30 text-to-CAD create-and-edit tasks, and 40 tasks that reconstruct mechanical parts from engineering drawings. Agents operate in a pinned FreeCAD environment and submit native CAD artifacts to a separate programmatic verifier. The primary metric is mean continuous task reward, combining geometry similarity and specification consistency rather than binary task success. A September 24, 2026 snapshot covers 13 model-agent systems; Claude Code with Claude Opus 5.5 achieves the highest overall point estimate, 61.03%, and an image-to-CAD mean reward of 84.66%. These measurements describe complete systems, not isolated model capabilities. We release versioned tasks, evaluation configurations, and inspectable result artifacts.
Abstract:Producing an executable CAD script is not sufficient to establish that an agent has satisfied an engineering design request. Evaluation must also consider the resulting geometry, specified dimensions, and native CAD modeling history. We present Parametric CAD Bench v3, a 100-task benchmark comprising 30 text-to-CAD creation tasks, 30 text-to-CAD create-and-edit tasks, and 40 tasks that reconstruct mechanical parts from engineering drawings. Agents operate in a pinned FreeCAD environment and submit native CAD artifacts to a separate programmatic verifier. The primary metric is mean continuous task reward, combining geometry similarity and specification consistency rather than binary task success. A September 24, 2026 snapshot covers 13 model-agent systems; Claude Code with Claude Opus 5.5 achieves the highest overall point estimate, 61.03%, and an image-to-CAD mean reward of 84.66%. These measurements describe complete systems, not isolated model capabilities. We release versioned tasks, evaluation configurations, and inspectable result artifacts.
31 pages, 12 figures, 11 tables. Benchmark: this https URL ; tasks and leaderboard: this https URL ; scoring code: this https URL
Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Graphics (cs.GR); Machine Learning (cs.LG)
arXiv:2611.02659 [cs.AI]
(or arXiv:2611.02659v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2611.02659 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 273 words
Related coverage
A nonlinear extension of parametric model embedding for dimensionality reduction in parametric shape design
arXiv · Jul 30, 2026
Draft: A Parametric Tool for Robot Design Exploration
arXiv · Oct 1, 2026
Analysis-Driven Procedural Generation of an Engine Sound Dataset with Embedded Control Annotations
arXiv · Jun 5, 2026
Kerr nonlinearity and three-wave mixing in superconducting resonators hosting Al-InAs weak links
arXiv · Aug 31, 2026