The Necessity of Setting Temperature in LLM-as-a-Judge

Abstract:Large Language Models (LLMs) are increasingly used as automated judges, yet decoding temperature is often set by intuition rather than evidence. We run over one million large scale evaluations, multiple open-source models, prompting strategies, and judging paradigms to examine how temperature affects judge reliability. Low temperatures yield consistent outputs with few errors, especially in smaller models, while higher temperatures reduce consistency and increase errors. Yet consistency alone does not guarantee sound judgments: low temperatures can narrow the range of reasoning explored. We characterize judge behavior through four reasoning personas and find that temperature shifts their distributions in model-dependent ways, most visibly in ambiguous cases where competing answers are hard to distinguish. Repeated high-temperature sampling reveals this hidden uncertainty, and combining it with majority voting or using high temperature for reasoning and low temperature for the final decision, improves agreement on ambiguous cases without hurting clear ones, at the cost of extra inference. Our findings suggest a simple rule: keep temperature low for stable outputs, but explore more when the judgment is hard.

Computation and Language (cs.CL)

arXiv:2603.28304 [cs.CL]

(or arXiv:2603.28304v3 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2603.28304 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.