You have a coding agent. It works, mostly. Then you tweak a prompt, swap a model, or add a tool, and now you have no idea whether you made it better or worse. It passed the one example you tried by hand, so you ship, and two days later it is confidently doing something dumb on a task you never thought to check.

This is the problem an LLM eval harness solves. Not "is the model good" in the abstract, but "did this change to my agent move the number up or down, on the work I actually care about." If you are building anything an agent will run more than a handful of times, the harness is the difference between engineering and guessing.

Here is how to build one that earns its keep, without pretending it does more than it does.

What an eval harness actually is

An LLM eval harness is three things: a set of cases, a set of graders, and a score you track over time. That is the whole idea. A case is a task with enough context to run it and a way to tell whether the result was acceptable. A grader turns one agent output into a verdict. The score aggregates those verdicts into a single number you can watch across runs.

The trap most teams fall into is treating "I ran it once and it looked right" as evaluation. It is not. A structured, schema-valid output can still be completely wrong, and a single hand-check cannot tell you if you are at 60% or 90%, or whether last week's fix quietly broke something else. A harness exists so that quality is a measurement, not an impression.

Start with cases you can trust

The instinct is to generate a big synthetic test set. Resist it. Twenty real tasks with known-good outcomes are worth more than five hundred invented ones, because the invented ones encode your assumptions, and your assumptions are exactly what the harness is supposed to check.

Pull cases from real work. For a coding agent, a case is usually a repository state, a task description, and some notion of what "done" looks like:

- id: fix-null-guard-42
  repo_fixture: fixtures/checkout-service
  task: "Add a null check to CartService.total so an empty cart returns 0"
  expect:
    files_touched: ["src/cart/CartService.ts"]
    tests_pass: ["cart.total.spec.ts"]

Keep the set small enough that you will actually maintain it, and biased toward the tasks that hurt when they go wrong. A good starting harness is 15 to 40 cases spanning your common task shapes: a simple fix, a multi-file change, something in a fragile area, something that should be refused because it is underspecified. Coverage of shape matters more than raw count.

Two kinds of graders, cheap ones first

A grader answers one question: was this output acceptable? You want two families of them, and you want to reach for them in the right order.

Deterministic graders are plain code with exact answers. They are fast, free, and they never have an opinion. Run these first and let them catch the obvious failures before you spend a cent on anything smarter:

def grade_deterministic(result, case):
    # the reference has to exist before anything else matters
    for path in result.files_touched:
        if not repo.exists(path):
            return Verdict.FAIL, f"touched nonexistent file {path}"
    if not result.compiles():
        return Verdict.FAIL, "did not compile"
    if not run_tests(case.expect.tests_pass).all_green():
        return Verdict.FAIL, "expected tests did not pass"
    return Verdict.PASS, "deterministic checks passed"

Does the file exist. Does it compile. Do the named tests pass. Is the JSON valid against the schema. These checks catch a surprising share of confident nonsense, and they cost nothing, so there is no reason to skip them.

Judge graders handle the things code cannot measure: is this the right approach, is the diff minimal, does the explanation match what the code does. Here you use a model to grade a model, which works, but only if you keep each judgment narrow and treat it as the expensive layer. There is real craft to doing this well without fooling yourself, enough that it is its own topic. The one rule to carry in now: ask the judge one specific question against one specific piece of evidence, never "is all of this good."

Grade the step and grade the run

A coding task is not one decision, it is a sequence: read some context, plan, edit, run tests, maybe fix and retry. If you only grade the final diff, you learn that something went wrong but not where. If you only grade the steps, you can have every step look reasonable and still end with a broken feature.

So grade both. At the step level, evaluate a single unit of work: was the plan sound, did this edit do what it claimed, was that the right file to touch. At the run level, evaluate the end-to-end outcome: did the feature actually get built and stay within scope. Step grades tell you which stage regressed when your number drops. Run grades tell you whether the whole thing delivered. You need both signals, because they fail independently.

Blocked is not failed

This is the distinction that separates a harness that helps from one that lies to you. Some cases the agent should not complete: the task was underspecified, a dependency was missing, the request was genuinely ambiguous. An agent that stops and says "I cannot do this without knowing X" is behaving correctly. If your harness scores that as a failure, you are training your metrics to reward guessing over honesty, which is the last thing you want from something that writes code.

Give your verdicts room for this:

class Verdict(Enum):
    PASS = "pass"
    FAIL = "fail"       # produced a wrong result
    BLOCKED = "blocked" # correctly could not proceed
    SKIP = "skip"       # not applicable to this run

A BLOCKED outcome is not a point against the agent. Track it separately, look at whether the blocks were legitimate, and only then decide if they point to a real gap. Folding "correctly refused" into "failed" will quietly push your agent toward confident wrong answers, because that is what your score is secretly rewarding.

Turn verdicts into one number you watch

The point of all this is a single score per run that you can trust and trend. A simple, defensible aggregation is the share of cases that passed, with partial credit for the ones that passed but took too many tries or too long:

suite_score = (fully_passed + 0.5 * passed_but_slow) / applicable_cases

where applicable_cases excludes the BLOCKED and SKIP results, so you are scoring what the agent was actually supposed to do. The exact formula matters less than picking one and holding it steady, because the value of the number is in the comparison across runs, not its absolute magnitude.

Then wire it into your loop. Run the harness on every meaningful change to the agent, its prompts, its tools, or its model. Store the score with the commit that produced it. When the number drops, you have a diff to blame and step grades to tell you where. This is the same discipline a test suite gives ordinary code, applied to a system whose behavior is probabilistic.

The honest limitation

An eval harness measures quality. It does not create it. A green suite is not a proof of correctness, it is evidence that the cases you thought to write still pass. There are three things it will not do for you, and you should know them going in.

Cases go stale. As your product changes, old cases stop reflecting real work, and a harness full of stale cases measures the past. Budget time to prune and add.

Judges have variance. A model grading a model is not a ruler, it is a noisy estimate. Keep judgments narrow, spot-check them against human review, and do not chase decimal points that are inside the noise.

And a suite score is a proxy, not the truth. Optimize it too hard and you will get an agent that is excellent at your harness and mediocre in the wild. The harness is a guardrail against regression, not a definition of good.

None of that makes it optional. It makes it honest. You are trading the illusion of certainty for a real, trackable signal, which is a very good trade for anything that ships code.

This is the discipline we build on inside Loopsfinity, where agents propose changes against a real codebase and a measured, repeatable evaluation sits between what an agent produces and what a human is asked to approve. We are not claiming a harness makes an agent trustworthy on its own. We are claiming that you cannot call an agent reliable if you cannot measure it, and measuring it is what a harness is for. If you are building your own, start with ten real cases and one honest number, and grow from there. For the wider picture of what "reliable" requires, see how to make an AI coding agent reliable on a real codebase.