Why You Shouldn’t Always Trust LLMs as Judges: Understanding Bias in Automated Evaluation

In the rush to automate evaluation, from grading student code to ranking research papers, we have embraced Large Language Models as judges. They are fast. These units are cheap. They scale. However, at a workshop at DHS 2026, Bhaskarjit Sarmah made a point that stuck with me: “you can’t trust LLM as a judge. I avoid using them.” LLMs pick up information from the web, which is filled with biased data created by humans and other biases that we will cover in this article. The different biases1. Position bias2. Verbosity bias3. Self-preference bias4. Entity and tone bias5. Identity bias6. Authority bias7. Bandwagon bias8. Distraction bias9. Fallacy-oversight bias 1. Position bias 2. Verbosity bias 3. Self-preference bias 4. Entity and tone bias 5. Identity bias 6. Authority bias 7. Bandwagon bias 8. Distraction bias 9. Fallacy-oversight bias Every bias we are going to explore has the same root cause: the judge answers from its own priors instead of from the evidence in front of it. Moreover, it learned from human-written text where longer answers signal effort, citations signal rigor, consensus signals correctness, and certain country and brand names carry decades of loaded coverage. Alignment training then rewarded whatever human raters liked,…