aiqbench analysis
This study demonstrates that automating code documentation for safety-critical domains like healthcare requires a robust multi-agent evaluation framework to mitigate significant performance gaps between different LLMs. By leveraging diverse model outputs and rigorous assessment across nine criteria, the approach effectively enhances documentation reliability while reducing manual effort. The findings underscore the necessity of advanced orchestration and prompt engineering to ensure context-aware, high-quality outputs in regulated environments.
LLM-Based Code Documentation Generation and Multi-Judge Evaluation
arXiv:2606.09852v1 Announce Type: cross Abstract: High-quality source code documentation is vital yet often neglected, especially in critical domains like healthcare where reliability and maintainability are essential. We presented an AI powered framework that automates documentation generation from code and repositories using eight state of the art Large Language Models (LLMs), including GPT, Gemini, Qwen, and LLaMA variants. Built on the PocketFlow orchestration framework, the system applies modular pipelines and advanced prompt engineering to produce structured, context aware documentation. To ensure quality and guide model selection, we introduced a MultiLLMasJudges evaluation framework, where four independent LLMs assess outputs across nine criteria, such as Completeness, Clarity, and Faithfulness. Experiments conducted on an open-source medical physics library, demonstrated showed a 42% performance gap between top and bottom models. By combining diverse model outputs, optimized prompting, and rigorous evaluation, our approach enhances documentation quality and reduces manual effort, especially in safety critical healthcare software.
Abstract:High-quality source code documentation is vital yet often neglected, especially in critical domains like healthcare where reliability and maintainability are essential. We presented an AI powered framework that automates documentation generation from code and repositories using eight state of the art Large Language Models (LLMs), including GPT, Gemini, Qwen, and LLaMA variants. Built on the PocketFlow orchestration framework, the system applies modular pipelines and advanced prompt engineering to produce structured, context aware documentation. To ensure quality and guide model selection, we introduced a MultiLLMasJudges evaluation framework, where four independent LLMs assess outputs across nine criteria, such as Completeness, Clarity, and Faithfulness. Experiments conducted on an open-source medical physics library, demonstrated showed a 42% performance gap between top and bottom models. By combining diverse model outputs, optimized prompting, and rigorous evaluation, our approach enhances documentation quality and reduces manual effort, especially in safety critical healthcare software.
ICAHS, \c{opyright} 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works
Human-Computer Interaction (cs.HC); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG); Multiagent Systems (cs.MA); Software Engineering (cs.SE)
arXiv:2606.09852 [cs.HC]
(or arXiv:2606.09852v1 [cs.HC] for this version)
Full text under the publisher's open licence · 232 words
Related coverage
Generate presentations with Amazon Bedrock AgentCore
AWS · Oct 14, 2026
How Boomi Scribe streamlines documentation using AWS
AWS · Sep 1, 2026
RAG Strategies for Natural Language-Based SQL Query and REST API Call Generation
arXiv · Aug 5, 2026
What is document AI?
Databricks · Jun 15, 2026