analiza aiqbench
This research introduces a novel method using Ising models to detect and adjust for correlations among LLM judges, ensuring that majority votes reflect independent evidence rather than shared biases or training lineages. By accounting for these dependencies, the approach significantly outperforms standard weighted voting baselines in evaluating retrieval-augmented generation systems. The findings highlight a critical nuance in AI evaluation: agreement among models does not automatically equate to accuracy if their decision-making processes are statistically dependent.
When LLM judges agree, should we believe them?
Discounting the opinions of LLM judges with highly correlated outputs ensures that panels of judges reflect a true diversity of perspectives.
Imagine evaluating a retrieval-augmented-generation system. A user asks a question, the system retrieves a text passage, and an LLM judge decides whether it’s relevant. To reduce noise, you ask several judge models to evaluate the same passage. Eight say “relevant”; two say “not relevant”. Eight out of 10 feels convincing. But the important question is not only how many judges agreed but how independently they arrived at that agreement. If the eight agreeing judges are genuinely different sources of evidence, then agreement is a strong signal. But if they share a prompt template, a training lineage, a model family, or a common blind spot, they may be repeating the…
Extras afișat conform termenilor RSS ai publicației · 109 cuvinte · textul integral și figurile sunt pe site-ul publicației
Subiecte conexe
Deconfusing Corrigibility
AI Alignment Forum · 17 oct. 2026
The current bottleneck is political will, not research
AI Alignment Forum · 11 iul. 2026
The basic argument for making AI better at moral philosophy
LessWrong · 14 oct. 2026
Why I Left Google DeepMind
LessWrong · 15 iul. 2026