Every distributed system has to answer one uncomfortable question: what happens when an action runs twice? For years that was a back-end concern, the kind of thing you handled in a payments service and forgot about elsewhere. Then the worker became an LLM, and the question moved to the front of the room, because agents retry far more than traditional code does, and they retry in messier ways.

An agent calls a tool, the call times out, and the agent tries again. A model returns malformed output, so the step reruns. A step reports low confidence, so the loop takes another pass. Public reports put tool-call retry rates somewhere in the 15 to 30 percent range for production agents (verify current figures for your own stack, they move). That is not an edge case. It is the normal operating mode of an agent, and it means every action your agent can take will eventually run more than once. Whether that is harmless or catastrophic is decided by one property: idempotency.

What idempotency actually means here

An operation is idempotent if running it twice produces the same result as running it once. Reading a file is idempotent. Setting a value to a fixed state is idempotent. Appending a row, sending an email, charging a card, or opening a pull request is not, because the second run adds a second effect.

The rule that follows is blunt and worth memorizing: a retry is only safe if the operation is idempotent, or if the workflow has a recorded result it can reuse instead of re-running. Retry a non-idempotent action and you do not get resilience, you get duplication. That is the whole game, and most agent reliability problems are some version of ignoring it.

The canonical failure makes it concrete. Agent A asks Agent B to charge a customer. B processes the charge but responds slowly. A times out, assumes failure, and retries. B, having no memory that this exact request already succeeded, charges the customer a second time. Every individual piece behaved correctly. The retry logic was textbook. The result is a double charge, because the operation was not idempotent and nothing deduplicated the second attempt.

Idempotency keys: the core technique

The standard fix is an idempotency key: a unique identifier attached to a unit of work, carried through every retry of that work, and checked before the action runs.

The pattern is check-before-execute. Before performing a side effect, the executor looks up the key in a store. If the key is present and the action already completed, it returns the stored result without doing anything. If the key is present but the action did not finish, it resumes or safely re-runs. If the key is absent, it executes, then records the key and the result. Retries reuse the same key, so the second attempt finds the first attempt's record and does nothing new.

The important design choice is where the key comes from. It has to be stable across retries and unique across distinct work, so derive it from the task, not from the attempt. A key like deploy:order-service:commit-abc123 identifies the work itself, so every retry of that deploy carries the same key. A key regenerated on each attempt defeats the entire mechanism, because the second attempt looks like new work.

Wrap the side-effecting steps of your agent in an executor that enforces this: key in, check the store, execute only if not already done, record the outcome. Once that executor exists, retries stop being dangerous and become boring, which is exactly what you want from your reliability layer.

Record and reuse: making a stubborn action safe

Some actions cannot be made naturally idempotent. You cannot un-send an email or un-post to an external system that has no dedup of its own. For those, the workflow carries the safety instead of the action.

The move is to record the result of the action durably the instant it succeeds, keyed by the idempotency key, before anything downstream can fail and trigger a replay. On any re-run, the workflow finds the recorded result and skips straight past the action to whatever came next. The side effect happened exactly once; the retry only replays the bookkeeping around it. This is the same principle behind resumability: a task that can be interrupted and resumed is one that records enough of its own progress to never redo committed work.

When even that is not enough, because the external system genuinely might see two requests, you push the dedup outward: pass your idempotency key to the downstream service if it supports one, so the duplicate is caught at the far end. Preventing the same unit of work from being dispatched twice in the first place is its own topic, covered in preventing duplicate work when a task fires twice.

A retry policy that knows when to stop

Idempotency makes retries safe. A retry policy decides whether they are useful. Retrying forever turns a transient failure into an expensive infinite loop, and it is one of the fastest ways to run up a bill.

A workable policy has three parts. Back off between attempts, so a struggling dependency gets room to recover instead of a thundering herd. Cap the attempts, because past a small number, more tries rarely succeed and mostly cost money. And classify failures: a timeout or a rate-limit is worth retrying, a validation error or a permission denial is not, because the same input will fail the same way. Retrying a deterministic failure is just paying to be told no twice.

When retries are exhausted, the task should stop and surface itself, not vanish. A dead-letter path or an escalation to a human turns a stuck task into a signal you can act on. Modeling those transitions explicitly, attempt, success, retry, exhausted, is much easier when the workflow is a state machine rather than a tangle of nested conditionals, and it interacts directly with how you handle concurrency and rate limits, since a rate-limited call should back off and requeue rather than count as a failure.

The honest limitation

Idempotency does not make an agent correct. It makes an agent's actions safe to repeat, which is a different and narrower promise. An idempotent operation can still be the wrong operation, executed reliably exactly once. Correctness is the job of review, testing, and verification, not of the retry layer.

There is also real cost to doing this well. Every side-effecting action needs a stable key, a store to check, and a durable record, and that machinery is work to build and to operate. Skip it and the system is simpler right up until the first double charge, at which point it is not simpler at all. The discipline is deciding, for each action, whether it is naturally idempotent, can be made safe by recording its result, or must push dedup to the far end, and then never letting an un-checked side effect ship.

This is a stance we build on in Loopsfinity: because agents retry and reality interrupts, side effects are designed to be safe to repeat, and a task that cannot finish stops and asks a human rather than looping. The mechanics of how we key and record that work are our own, but the principle is not, and it is one any team running agents against real systems should hold. It is one piece of the larger picture in AI agent architecture, where reliability, design, and scale meet.