The first time you build an evaluation for a coding agent, you almost always start with two buckets: it passed, or it failed. It is a natural place to begin, and it is quietly wrong in a way that will cost you later. Because there are two very different ways for an agent to not-pass, and if you file them in the same drawer, you end up optimizing for the wrong behavior without ever deciding to.
An agent can be wrong: it proceeded, produced an answer, and the answer was bad. It edited the wrong function, broke a caller, misread the requirement. That is a real failure, and you want your evaluation to catch it.
An agent can also be blocked: it could not proceed honestly. The task was underspecified, a dependency was missing, a required piece of context was not available, the instructions contradicted each other. A good agent in that situation stops and says so. That is not a failure. That is the behavior you want.
Collapse those two into a single "fail" and you have built a scoreboard that cannot tell the difference between an agent that broke your code and an agent that refused to guess. Over time, that scoreboard will push you toward the first kind.
Why the distinction is not pedantic
Consider what happens when you actually use a pass/fail metric to make decisions, which is the entire point of having one. You tune prompts against it. You pick models against it. You decide whether a change to the pipeline helped or hurt against it.
Now put two agents in front of that metric on an underspecified task. Agent A recognizes the ambiguity and stops: "I cannot implement this safely without knowing which of the two auth flows you mean." Agent B does not hesitate. It picks one, writes confident code, and moves on. Half the time it guesses right.
Under pass/fail, Agent A failed every time. Agent B passed half the time. Your metric now prefers Agent B. So every optimization you run, every prompt you tune, every model you select, is nudged toward the agent that bluffs. You are not choosing to reward guessing. Your evaluation is choosing it for you, silently, and for anything that ships code, the bluffer is the dangerous one. A blocked task costs you a question. A wrong task that slipped through because it looked confident costs you an incident.
Blocked is a signal, not a defect
The healthier model treats an agent's outcomes as at least three states, not two. It passed. It was wrong. Or it was blocked, and blocked is routed somewhere useful rather than counted as a loss.
That reframing changes what the outcome means downstream. A wrong result is a defect to fix in the agent or its context. A blocked result is a message to a human: the task was not ready, and here is what is missing. The right response to blocked is not "the agent failed," it is "the agent found a gap in the input before that gap became a bug." Handled well, blocked becomes one of the most valuable things an agent produces, because it surfaces the ambiguity while it is still cheap to resolve.
This is the same idea that runs through structured output guarantees the shape, not the truth, from a different angle. There, the point is that a well-formed answer can still be false. Here, the point is that a refusal to answer can be exactly right. Both are cases where the surface signal (valid JSON, or "did not complete") tells you nothing about whether the agent did the correct thing.
Telling one from the other
The line between blocked and wrong is usually clean if you look at what the agent actually did. Did it act and produce an output that turned out to be incorrect? That is wrong. Did it decline to act, and give a reason a competent engineer would accept? That is blocked. The tell is the reason. A blocked result comes with a specific, checkable "I could not do this because X." A wrong result comes with a confident answer that does not survive contact with your tests.
Which is also why blocked and your other reliability machinery reinforce each other. When you have acceptance criteria an agent can actually test against, "blocked" often means "I cannot satisfy these criteria with what I was given," which is precise and actionable. When the agent reads your tests before it writes code, it can often detect the blocker early, before it wastes a full attempt. Good grounding does not just reduce wrong answers. It makes the blocked ones arrive sooner and carry better reasons.
The honest limitation
Splitting blocked from wrong only helps if you are honest about the split, and that is harder than it sounds. Models are not perfectly calibrated. Some of what an agent reports as "blocked" is a cop-out on a task it simply found hard, and some of what looks like a clean "wrong" was avoidable with better context. If you let "blocked" become a no-questions-asked escape hatch, agents will learn to reach for it, and your reliability numbers will look great while nothing ships. So blocked has to be audited: you read the reasons, you confirm the blocker was real, and you feed the pattern back into better inputs. And in the end a human still decides what counts as a legitimate reason to stop, because that judgment is not something you want to fully delegate to the thing being judged.
What you get for that work is an evaluation that rewards the behavior you actually want: an agent that is right when it can be, and honest when it cannot. That honesty is not a consolation prize. On a real codebase it is the difference between an agent you can leave running and one you have to watch.
This is a distinction we take seriously in building agents that are reliable on a real codebase. At Loopsfinity a blocked result is treated as a signal to raise with a human or to re-plan the work, not a failure to paper over, because an agent that knows the edge of what it can do safely is worth more than one that never admits there is an edge. How we categorize outcomes internally is our own business. The principle, that blocked and wrong are different and your evaluation should say so, is one worth adopting whatever you are building.