aiqbench analysis
This research introduces a unified LLM-as-a-judge framework to rigorously measure perturbation strength, addressing a critical gap in evaluating the self-consistency of large language model explanations. By controlling for input versus chain-of-thought perturbations, the study demonstrates that fairness in consistency judgments depends on maintaining consistent perturbation types across comparisons. These findings provide essential methodological clarity for researchers developing robust evaluation metrics and testing model reliability under varying noise conditions.
Enhancing Assessment of Self-Consistency in LLM Explanations using Perturbation Strength
arXiv:2609.30849v1 Announce Type: new Abstract: Prior work has examined the self-consistency of LLM-generated explanations using surface-level perturbation methods. However, the strength of these perturbations is not explicitly measured and controlled. In this work, we propose an LLM-as-a-judge approach to measure perturbation strength in a unified manner across input and CoT perturbations. We then evaluate the self-consistency in explanations generated from various LLMs under controlled strength conditions, ensuring a fair comparison across perturbation types. Experiments show that our proposed LLM-based perturbation strength measure outperforms other embedding- and probability-based approaches and that input perturbations generally affect LLMs more strongly than CoT perturbations. Our work suggests that judgments about a model's self-consistency is fair only within the same perturbation type.
Abstract:Prior work has examined the self-consistency of LLM-generated explanations using surface-level perturbation methods. However, the strength of these perturbations is not explicitly measured and controlled. In this work, we propose an LLM-as-a-judge approach to measure perturbation strength in a unified manner across input and CoT perturbations. We then evaluate the self-consistency in explanations generated from various LLMs under controlled strength conditions, ensuring a fair comparison across perturbation types. Experiments show that our proposed LLM-based perturbation strength measure outperforms other embedding- and probability-based approaches and that input perturbations generally affect LLMs more strongly than CoT perturbations. Our work suggests that judgments about a model's self-consistency is fair only within the same perturbation type.
Computation and Language (cs.CL)
arXiv:2609.30849 [cs.CL]
(or arXiv:2609.30849v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.30849 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 207 words
Related coverage
Drivers predict self-driving car actions better with AI insights
Quantum Zeitgeist · Sep 2, 2026
Building and evaluating model diffing agents
AI Alignment Forum · Jun 12, 2026
Informe de riesgos de la IA de frontera (febrero–marzo de 2026)
METR · May 19, 2026
Frontier AI Risk Report (February–March 2026)
METR · May 19, 2026