aiqbench analysis
SafeLLM4SE addresses critical reproducibility issues in LLM-based software engineering by introducing a statistical framework that treats model outputs as stochastic processes rather than deterministic artifacts. This methodology provides essential tools for calculating confidence intervals and distinguishing between actual quality improvements and estimation uncertainty, ensuring more reliable benchmark comparisons. The accompanying open-source package enables researchers to adopt these rigorous evaluation standards immediately.
SafeLLM4SE: Statistical Evaluation and Reporting for LLM-based Software Engineering Systems
arXiv:2609.37294v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used for software engineering tasks, yet their stochastic behavior challenges the validity, reproducibility, and comparability of their evaluations. Conventional practices such as reporting a single output, an average score, best-of-N, or pass@k performance can obscure variability and estimation uncertainty, potentially leading to misleading conclusions about system reliability. This article presents SafeLLM4SE, a practical methodology and reporting standard for statistically principled evaluation of LLM-based software engineering systems. Rather than treating generated outputs as deterministic artifacts, SafeLLM4SE treats them as realizations of a stochastic process and distinguishes quality, stability, and estimation uncertainty. It combines adaptive sampling with confidence intervals, distribution-aware statistical comparisons, effect sizes, and a minimum reporting standard covering model configuration, reproducibility, evaluation procedures, and resource usage. SafeLLM4SE is also provided as an open-source software package available on PyPI, enabling researchers and practitioners to reproduce and extend the methodology. We illustrate its application by comparing two LLMs on HumanEval, a benchmark of programming problems assessed through functional tests.
Abstract:Large language models (LLMs) are increasingly used for software engineering tasks, yet their stochastic behavior challenges the validity, reproducibility, and comparability of their evaluations. Conventional practices such as reporting a single output, an average score, best-of-N, or pass@k performance can obscure variability and estimation uncertainty, potentially leading to misleading conclusions about system reliability. This article presents SafeLLM4SE, a practical methodology and reporting standard for statistically principled evaluation of LLM-based software engineering systems. Rather than treating generated outputs as deterministic artifacts, SafeLLM4SE treats them as realizations of a stochastic process and distinguishes quality, stability, and estimation uncertainty. It combines adaptive sampling with confidence intervals, distribution-aware statistical comparisons, effect sizes, and a minimum reporting standard covering model configuration, reproducibility, evaluation procedures, and resource usage. SafeLLM4SE is also provided as an open-source software package available on PyPI, enabling researchers and practitioners to reproduce and extend the methodology. We illustrate its application by comparing two LLMs on HumanEval, a benchmark of programming problems assessed through functional tests.
This work has been submitted for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible
Software Engineering (cs.SE); Machine Learning (cs.LG)
arXiv:2609.37294 [cs.SE]
(or arXiv:2609.37294v1 [cs.SE] for this version)
https://doi.org/10.48550/arXiv.2609.37294 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 207 words
