An agent that works on a real codebase carries a lot of baggage into every call. The system instructions, the conventions it has to follow, the slice of the repo it needs to reason about: all of that gets sent to the model before the actual task even begins. For a grounded agent, that context is often the majority of the input tokens, and it barely changes from one call to the next.

That is the exact shape prompt caching is built for, and it is the closest thing to free money in the whole cost model. If you are running an agent against your own code and you are not caching, you are paying full price to re-read the same context over and over.

What prompt caching actually charges for

Every major provider now offers prompt caching, and the economics are similar across them: the first time the model sees a chunk of context, you pay to write it to the cache, and every subsequent call that reuses that exact prefix reads it at a steep discount. The cached portion typically costs around a tenth of the normal input rate, roughly a 90 percent reduction on those tokens. (Rates and the exact discount move, so verify the current numbers with your provider before you model it.)

The saving only applies to the prefix that is identical, byte for byte, to what was cached. That is the whole game. Cache what repeats, and structure your prompt so as much as possible repeats.

Structure the prompt so the stable part caches

The single most important habit is ordering. Put the stable, reusable material first and the volatile, per-task material last:

  • Stable prefix (cache this): system instructions, coding conventions, the durable slice of codebase context the agent needs, anything that holds across many calls.
  • Volatile suffix (do not expect to cache this): the specific task, the current file diff, the latest tool output, timestamps.

Caching keys on an exact prefix match, so a single changed character early in the prompt invalidates everything after it. The classic own-goal is putting something trivial and volatile near the top: a current timestamp, a per-request id, a reordered list of files. It looks harmless and it silently busts the cache on every call, so you write to the cache constantly and read from it never. Keep the top of the prompt boring and identical, and let the churn live at the bottom.

There is also a time dimension. Caches have a lifetime, and an entry that is not reused within that window expires and has to be written again. Caching pays off when calls that share a prefix happen close together, which is exactly what an agent working through one task does.

When it helps, and when it does not

Prompt caching is not a universal win. It pays when two things are true: the prefix is large, and it is reused many times before it expires.

  • Big win: a repo-grounded agent making a sequence of calls against the same context (plan, implement, test, check). The context is large and stable, the calls are close together, and the cache hit rate is high.
  • Little or no win: one-off calls with no repetition, or a workload where the context changes substantially every time. There is nothing stable to amortize, so you mostly pay the write cost with few reads to recoup it.

This is why caching matters more for agents than for chatbots. A chat turn often has a small, shifting context. A coding agent drags a big, stable brief through many steps, which is the ideal cache profile.

What it does to the per-feature bill

Shipping a feature is many small runs, not one big call, and those runs share most of their context. Without caching, you pay full input price for that shared context on every run in the sequence. With caching, you pay it near-full once and at a tenth thereafter. Because input can be the bulk of a grounded agent's tokens, the effect on the total is large, often the difference between a workload that is comfortable to run continuously and one you ration.

Caching also composes with the other levers rather than competing with them. It stacks cleanly with model routing (route the cheap steps to a cheap model and cache the shared prefix on both), with batching for the latency-insensitive work, and with spend caps that bound the tail. None of these cancel each other out; they add up.

The honest limitation

Caching lowers the price of context you were going to send anyway. It does not reduce how much context you send, and it is not a substitute for sending less. If your prefix is bloated, caching just makes the bloat cheaper, not right; a tighter, more relevant context is still the better first move, and caching is what you do after that.

A few other caveats worth stating plainly. The first write to the cache can cost slightly more than a normal call, so caching only nets out if you get enough reads before the entry expires; a low hit rate can leave you worse off. The discount applies to the cached input only, not to output, so verbose generations stay expensive no matter how well you cache. And the exact discount, cache lifetime, and minimum cacheable size differ by provider and change over time, so treat any specific percentage as a figure to re-verify, not a constant.

Used well, though, it is the highest-leverage cost move available to anyone running an agent on a real repository, precisely because a grounded agent's prefix is so large and so stable.

That profile is why we lean on it hard at Loopsfinity. The context an agent needs is a big, stable prefix reused across a task's runs, which is the best possible case for caching, so we treat cache-friendly prompt structure as a first-class concern. The specifics of how we assemble and cache that context are ours; the principle, and the ordering habit that earns the discount, are yours to apply to whatever you are building.