At some point every team building with agents hits the same wall. You have thousands of outputs to evaluate, a plan here, a summary there, a diff explanation, an answer to a support question, and no human has time to read them all. So you do the obvious thing: you ask a model to grade the model. That is LLM-as-a-judge, and it is one of the most useful and most quietly abused techniques in the whole agent toolkit.
Useful, because it scales evaluation to a level humans cannot match. Abused, because it is easy to wire up a judge, watch it emit confident scores, and never check whether those scores mean anything. A judge that is wrong in a consistent direction is worse than no judge, because it gives you a number to trust while pointing you the wrong way.
This is a guide to using LLM-as-a-judge without fooling yourself: what it is good at, the specific ways it fails, and how to build a judge you can actually rely on.
What LLM-as-a-judge actually means
The pattern is simple. Instead of a human rating an output, you give a model the output (and sometimes a reference answer) and ask it to score or classify. "Is this summary faithful to the source, yes or no." "Rate this explanation for clarity from 1 to 5." "Does this response satisfy the acceptance criteria." The judge returns a verdict, and you aggregate verdicts across a whole test set.
It fills a real gap. Deterministic checks can tell you whether JSON parsed or whether a referenced file exists, but they cannot tell you whether a plan is coherent or an explanation is actually helpful. Those are judgment calls, and until recently the only thing that could make them at scale was a person. A model doing model grading is the first tool that makes subjective evaluation cheap enough to run on every change.
Where it works well
LLM-as-a-judge earns its keep on narrow, well-defined questions where a human would agree with itself run to run. "Does this answer contradict the provided context." "Is this response in the right format and tone." "Which of these two outputs better follows the instruction." Pairwise comparison in particular tends to be more stable than absolute scoring, because "A is better than B" is an easier judgment than "B is a 7 out of 10."
It also works well as a filter rather than a final grade. Using an automated eval judge to flag the bottom ten percent of outputs for human review is a very different bar than trusting it to certify the top. The first use survives a noisy judge. The second does not.
Where it fails
Here is the part that gets skipped. A judge is a language model, so it inherits every failure mode of a language model, and a few extra that come from the grading setup itself.
Position bias. Ask a model which of two answers is better and it will often favor whichever one you showed first (or last), independent of quality. If your harness always puts the candidate before the reference, that bias bakes straight into your scores.
Verbosity bias. Judges tend to reward longer, more elaborate answers even when the shorter one is more correct. An agent that learns to pad its output can climb your eval without getting better, which is the opposite of what you want.
Self-preference. A model asked to judge outputs tends to rate text from its own family more highly. If your judge and your generator are the same model, you have a grader with a soft spot for the student it is grading.
Inconsistency. Run the same judgment twice and you will not always get the same verdict. Temperature, phrasing, and ordering all move the result. A judge that flips on reruns cannot anchor a quality metric you plan to track over time.
Gaming. This is the dangerous one. The moment a judge's score becomes a target, the system under test starts optimizing for the judge rather than for the truth. Outputs learn to say the words the judge rewards. The metric goes up while quality stays flat or drops. Any judge you optimize against will eventually be gamed unless you actively guard against it.
None of these are reasons to abandon the technique. They are reasons to design around it.
How to make a judge you can trust
The reliable version of LLM-as-a-judge looks different from the naive one in a few specific ways.
Judge one dimension at a time. A prompt that asks "is this good" invites the model to blend a dozen fuzzy factors into one soft number. Split it. One judgment for faithfulness, one for format, one for whether it meets the stated criteria. Narrow single-dimension calls are more consistent, easier to debug, and cheaper to rerun. The related idea from structured output validation applies here too: precise questions get precise answers.
Give it a rubric, not a vibe. "Rate clarity 1 to 5" leaves the model to invent its own scale. Instead, spell out what each verdict means: what counts as a pass, what specific thing makes it a fail, with a short example of each. A judge anchored to explicit criteria agrees with itself far more often than one asked for a gut score.
Ask for a reason before the verdict. Requiring the judge to state its evidence first, then its verdict, both improves the judgment and gives you something to audit. When a score looks wrong, the reason tells you whether the judge misread the output or you wrote a bad rubric.
Use cheap models for narrow checks. A tightly scoped pass or fail judgment does not need your most expensive model. Small, fast models do narrow classification well, which matters when you are running the judge across a large test set on every change. Save the expensive model for the genuinely hard calls.
Control for the known biases. Randomize the order of candidates to cancel position bias. Cap or normalize for length to blunt verbosity bias. Where you can, use a different model family for the judge than for the generator to reduce self-preference. These are small changes to the harness that remove whole classes of error.
Calibrate against humans, then keep checking. Before you trust a judge, have a person label a sample and measure how often the judge agrees. That agreement rate is the real credibility of your metric. And it is not a one-time step: recalibrate periodically, because a judge that was accurate last quarter can drift as your outputs change. Run the judge inside a real eval harness so this calibration is part of the pipeline, not a thing you do once and forget.
The honest limitation
Even a well-built judge is an estimate, not ground truth. It reduces the cost of evaluation and it catches a lot, but it does not remove the need for human judgment on the decisions that matter. The right way to think about LLM-as-a-judge is as a way to spend human attention well: let the judge triage the volume so that people look at the cases where being wrong is expensive, instead of reading everything or reading nothing.
That is roughly how we treat automated grading inside Loopsfinity. Model-graded checks help sort and surface work, but the decisions that carry real consequences still route to a person. A judge is a smoke detector, not a fire marshal. It is very good at telling you where to look, and you should never let it decide, on its own, that the building is safe.
Used that way, model grading model is one of the highest-leverage tools you have. Used as an unchecked oracle, it is a confident number that quietly tells you what you want to hear. The difference is entirely in how you build it.