LLM-as-a-Judge: Score AI Agent Outputs Automatically

An agent can pass every deterministic test and still give a poor answer. A support agent can call the correct order lookup tool, retrieve the right policy, and then leave the refund window out of its response. The tool assertions pass, but nothing checked whether the final answer was accurate, complete, and useful. LLM-as-a-judge evaluation covers that gap. A second model reviews the candidate agent’s output against criteria you write in plain language and returns a score. This gives you a repeatable way to test open-ended responses that have several valid wordings. You run a candidate, then give its output and any relevant tool results to a judge. The judge scores that evidence against a rubric you wrote. If the score falls below the threshold you set, the evaluation fails. The judge is a model call like any other. It reads text, applies your criteria, and returns a number. Treat that number as a measurement taken with a fixed setup, not as ground truth. The candidate is the system under test. It answers the user, calls tools, and writes the output you care about. The judge is a separate model. It does not solve the original task. It only grades…