A compiler error is a gift. It is loud, it is specific, it stops you before the mistake reaches anyone, and it costs you a few minutes. The failure mode that actually hurts with AI-written code is the opposite of that. It is code that compiles, passes a quick read, looks exactly like what a competent engineer would write, and is wrong anyway. It handles the happy path and drops the edge case. It calls the right function with a subtly wrong argument. It implements the feature the ticket seemed to ask for rather than the one it meant. This is plausible-but-wrong output, and it is the most expensive thing an AI coding agent produces.
The expense is not in the writing. The tokens are cheap. The expense is in the finding, because plausible-but-wrong code is engineered, by the very nature of how these models work, to survive the checks that catch obvious errors.
Why models fail plausibly and not obviously
A language model is trained to produce output that looks like the correct answer. That is the whole objective, and most of the time looking correct and being correct coincide. The problem is the gap between them, and the gap is exactly where the plausible-but-wrong failure lives.
A model does not have a concept of "I do not know" that reliably fires. Faced with uncertainty, it does not stop and flag the gap the way a careful engineer would. It produces the most probable continuation, which is a fluent, confident, well-structured answer whether or not it is true. The result is that the model's failures inherit its fluency. It does not fail like a beginner who writes obviously clumsy code. It fails like an expert having a bad day, producing something that reads authoritatively and is wrong in one specific place you have to hunt for.
That is why plausible-but-wrong is worse than obviously-broken. Obviously-broken announces itself. Plausible-but-wrong hides inside code that passes every superficial test of quality, including the most dangerous one: it looks like something you would have written.
The cost is asymmetric and it lands late
The reason this failure mode dominates the cost of AI-assisted work is that its cost is asymmetric and back-loaded.
An obvious error is caught in seconds by the person or the tool nearest to it. A plausible error slips the first reviewer, because it looks fine. It slips the tests, because the tests cover the cases someone thought of and the bug is in a case nobody did. It merges. It deploys. And then it surfaces as a production incident, a corrupted record, or a customer complaint, at which point the cost includes not just the fix but the investigation, the blast radius, and the erosion of trust in every other change the agent made. A bug that would have cost minutes at the point of writing costs hours or days at the point of discovery.
This asymmetry is what makes "the agent is fast" a misleading measure. Speed to a plausible-but-wrong diff is not speed to a working feature. If the output has to be unwound later, the agent did not save time, it borrowed it at a high interest rate. That hidden borrowing has a name and a whole economics of its own, which we cover in the rework tax.
Why validation is not verification
The instinct, when you realize the agent can be confidently wrong, is to add checks. That instinct is right, but the most common version of it stops one step too early.
Structured output and schema validation are genuinely useful and you should use them. They guarantee the shape of the answer: the JSON parses, the fields are present, the enums are legal. But shape is not truth. An agent can return perfectly valid, schema-conformant JSON that claims a function lives in a file where it does not, or that describes a change it did not actually make. The format is impeccable and the content is a lie. We pull this distinction apart in structured output guarantees the shape, not the truth, because conflating the two is how teams build a validation layer that feels rigorous and catches none of the failures that matter.
Verification is the harder, more valuable thing: checking that the content is actually true, not just well-formed.
Catching it: cheap checks first, judgment second
The good news is that a surprising share of plausible-but-wrong output is catchable by cheap, deterministic code, long before you reach for anything expensive.
If the agent names a file, check the file exists. If it references a symbol, look for the symbol. If it claims one module imports another, read the imports. These checks are fast, free, boring, and they catch a real fraction of confident hallucinations at the point of writing, where the fix is a retry rather than an incident. The highest-value ones are worth building first, and we work through them in catching hallucinated file paths cheaply.
What deterministic checks cannot assess is judgment: whether a plan is coherent, whether a change actually satisfies the intent behind a ticket, whether a summary reflects the diff it claims to. For those you need evaluation as a standing practice, built in layers so cheap graders handle everything with a right answer and model-based judgment handles only what code cannot. Standing that up is the subject of building an eval harness for a coding agent. The combination, cheap deterministic checks plus narrow model-based judgment, is what turns "it looked fine" into "it passed the checks that define fine."
The honest limitation
No amount of checking makes plausible-but-wrong output impossible, and pretending otherwise just moves the false confidence from the model to your test suite. Deterministic checks only catch the errors you thought to check for. Model-based judgment has its own failure modes and can be plausibly wrong in exactly the way it is meant to catch. A determined subtle bug, in a case no test covers and no checker anticipated, can still reach production.
What checking changes is the economics and the timing. It moves the cost of a wrong answer from late and large to early and small, from a production incident to a failed check the agent has to satisfy before it proceeds. That is the realistic win: not eliminating the failure mode, but making it cheap and visible instead of expensive and hidden.
Where this leaves you
Plausible-but-wrong is the failure mode that makes AI coding feel risky, and the feeling is justified, because the output is designed by its nature to survive a glance. The answer is not to trust the fluency and it is not to distrust the tool entirely. It is to verify truth and not just shape, with cheap checks at the point of writing and real evaluation over time.
This is why we treat checking as a first-class part of the loop at Loopsfinity rather than a step bolted on at the end: an agent's output is not done because it looks done, it is done because it passed the checks that define done. The demo-to-production version of this problem is covered in the demo-to-production cliff, the way it worsens on big systems in context rot, and the full map of failure modes in why AI coding agents fail in production.