aiqbench analysis
This research demonstrates that applying natural language uncertainty estimation methods directly to code generation is insufficient due to code's unique properties like token fragility and executability. By introducing a three-axis framework that accounts for lexical, algorithmic, and functional differences, the study achieves an 8.1-point AUROC improvement over existing baselines while offering a low-cost alternative for specific models. The findings underscore the necessity of code-specific designs to mitigate safety risks in LLM-driven software development.
Code Is More Than Text: Uncertainty Estimation for Code Generation
arXiv:2606.09577v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed as code generators, where silently wrong programs pose real safety and reliability risks. Reliable uncertainty estimation (UE) is essential for selective prediction, human-in-the-loop review, and downstream agentic decisions. Yet most existing code UE methods are inherited from natural language (NL) generation and ignore properties that make code distinct. We argue that code differs from NL in three ways: a single wrong token can break an entire program (token fragility); algorithmic intent and concrete implementation can disagree independently (intent-code gap); and programs can be executed (executability). We instantiate these properties as three orthogonal uncertainty axes: lexical (Top-K token entropy), algorithmic (pseudo-code consistency), and functional (behavioral consistency). Across five code LLMs, our three-axis ensemble improves average AUROC from 0.696 for the strongest NL-derived baseline to 0.776 (+8.1 points). Notably, on Qwen3-14B, our single-pass Top-K token entropy matches the strongest multi-pass baseline while being over 3x cheaper; across models, it remains a competitive low-cost signal. These results suggest that code UE deserves code-specific design rather than direct NL ports.
Abstract:Large language models (LLMs) are increasingly deployed as code generators, where silently wrong programs pose real safety and reliability risks. Reliable uncertainty estimation (UE) is essential for selective prediction, human-in-the-loop review, and downstream agentic decisions. Yet most existing code UE methods are inherited from natural language (NL) generation and ignore properties that make code distinct. We argue that code differs from NL in three ways: a single wrong token can break an entire program (token fragility); algorithmic intent and concrete implementation can disagree independently (intent-code gap); and programs can be executed (executability). We instantiate these properties as three orthogonal uncertainty axes: lexical (Top-K token entropy), algorithmic (pseudo-code consistency), and functional (behavioral consistency). Across five code LLMs, our three-axis ensemble improves average AUROC from 0.696 for the strongest NL-derived baseline to 0.776 (+8.1 points). Notably, on Qwen3-14B, our single-pass Top-K token entropy matches the strongest multi-pass baseline while being over 3x cheaper; across models, it remains a competitive low-cost signal. These results suggest that code UE deserves code-specific design rather than direct NL ports.
Computation and Language (cs.CL); Machine Learning (cs.LG); Software Engineering (cs.SE)
arXiv:2606.09577 [cs.CL]
(or arXiv:2606.09577v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.09577 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 274 words
Related coverage
Explainable Uncertainty Estimation for Reliable Medical AI
arXiv · Sep 1, 2026
A Unified Risk View of Uncertainty: Posterior Risk for Disentanglement and Evaluation Beyond Proxies
arXiv · Aug 7, 2026
EviDep: Uncertainty-Aware Multimodal Depression Estimation via Disentangled Evidential Learning
arXiv · Sep 16, 2026
Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models
arXiv · Sep 24, 2026