Idempotency Patterns for Agent Tool Calls With Side Effects

Agent tool calls that touch the outside world (charging a card, sending an email, writing an order to a database) inherit a problem distributed systems engineers have wrestled with for decades: retries that assume an operation is idempotent when it isn't. The standard agent loop treats a failed tool call the way a browser treats a failed page load, just try again. That assumption held together reasonably well when agents made one tool call per task. It does not hold up now, and the gap between what agent frameworks assume and what production systems actually need is where duplicate charges, ghost orders, and fourteen-email onboarding sequences come from.

How the retry surface grew 44× in a single year

LangChain's 2024 telemetry puts a number on how fast this surface expanded. Tool calls went from 0.5% of agent traces in 2023 to 21.9% in 2024, and average steps per agent run climbed from under one to 7.7. Tool calls went from 0.5% of agent traces in 2023 to 21.9% in 2024, and average steps per agent run climbed from under one to 7.7, so the retry surface grew roughly 44× in twelve months. That is not a gradual trend line, but a phase change in what agents are actually doing.

Each of those 7.7 steps is a candidate for a non-idempotent side effect, such as an email send, a charge, a database write, or a downstream API call that itself triggers further calls. The compounding effect is not linear, either. A five-step chain where each step has some retry probability does not fail five times as often as a one-step call; the failure modes multiply against each other, because a retry at step four can re-trigger consequences from steps one through three that already landed.

None of this is happening against a delivery layer designed for exactly-once semantics. Stripe retries webhooks for up to three days. AWS SQS standard queues document duplicate delivery as part of the contract, not a bug. HTTP retries are the norm across the stack the agent sits on top of. The agent inherited at-least-once delivery from infrastructure that was built assuming the application layer would handle deduplication, and in most current agent frameworks, nothing does.

Gartner's forecast puts a ceiling and a floor on how seriously to take this. Forty percent of enterprise applications are expected to ship task-specific agents by the end of 2026. In the same forecast, more than 40% of agentic AI projects are expected to be cancelled by the end of 2027, and the reason cited is reliability and governance gaps, not model capability. The problem is accelerating rather than stabilizing as the technology matures. It's accelerating, because more steps per agent and more agents in production both push in the same direction.

Five production failure modes that all trace back to the same root cause

Post-mortems on agent incidents tend to open with some version of "the agent acted weirdly." The agent did what the framework told it to do. Retry on timeout, up to three times. Call the tool again if the tool call failed. Generate an order ID. Every one of these instructions was followed correctly. The failures below are engineering failures in the retry contract, not behavioral anomalies in the model.

The fourteen-email incident is the cleanest case of framework defaults compounding. A B2C signup agent wrapped an internal API that was eventually consistent: it returned a 202 before the message was actually enqueued. Under load, that 202 sometimes arrived after a timeout the agent had already given up waiting for: the message was enqueued even though the agent believed the call had failed. The framework's default was retry-on-timeout, up to three attempts. A separate downstream agent, watching for "incomplete onboarding" states, fired four sequential re-triggers on top of that. Nine minutes later, one mailbox had fourteen emails.

The double subscription charge is subtler, because the Stripe call was actually built correctly. It used Stripe's idempotency key with a 24-hour deduplication window, exactly as documented. The problem sat one layer downstream: the internal entitlement-grant call that ran after the charge had no idempotency protection of its own. When the agent retried the full sequence, the Stripe call correctly deduplicated and did nothing on the second pass, but the entitlement grant executed twice. The lesson generalizes past this one incident: idempotency is not a property you can attach to a single call and consider solved. It has to hold at every layer the retry can reach.

The ghost order is the incident that exposes the real fault line, and it deserves treatment on its own.

The ghost order

The order management system in question was, in fact, idempotent on client-supplied order ID. That part of the architecture worked exactly as designed. The failure was upstream of it: the agent's prompt instructed it to "generate an order ID," rather than to reuse the same order ID across retries of the same logical request. Because the instruction left the ID generation to the model on each call, and the model has no persistent memory of what it generated last time absent explicit state, it produced a fresh identifier on every retry. The OMS dutifully treated each one as a new, valid order, and every individual system layer had done what idempotency design calls for.

Why LLM nondeterminism breaks the foundational assumption existing idempotency tools rely on

Every idempotency mechanism in distributed systems, whether it's a unique request ID, a recorded nondeterministic value, or deterministic logic in the calling client, rests on one shared assumption: the caller sends an identical request when it retries. Deduplication logic on the receiving end works by comparing the new request against the old one and recognizing a match. That comparison is the entire mechanism. Without the guarantee of a matching request, the mechanism has nothing to compare against.

LLM agents violate that assumption at a level below the reasoning most engineers plan around. Even with deterministic settings, LLM inference can produce different token sequences across runs of what is nominally the same prompt. That's before accounting for the more obvious source of variation: an agent asked to "generate an order ID" or "compose a follow-up email" is, by design, synthesizing a fresh response each time, not recalling a cached one.

The practical consequence is that a restored or retried agent generates a request the downstream server accepts as new, because the reference ID, a parameter value, or the ordering of fields differs from the original attempt just enough that no deduplication logic fires. The server isn't malfunctioning. It rejects exact duplicates and accepts anything that looks different, which is what it was built to do. The ghost order incident is the clean illustration of this, because the OMS had correct, working idempotency support, and the LLM's nondeterminism defeated it anyway. The fix was never going to live inside the OMS. It has to live in how the request gets constructed before the OMS ever sees it.

Pattern 1: Idempotency keys generated by the orchestrator, not the model

The rule that follows directly from the ghost order incident: any tool with a side effect requires an idempotency key, and that key has to be generated in the orchestrator layer, never by the model. Asking the model to generate its own idempotency key is asking the least deterministic part of the system to produce the one value that has to stay constant across retries.

Keys need to be derived from things that don't change on retry, such as the workflow run ID, the step index within that run, and the action type being performed. A timestamp and a UUID minted fresh at call time both vary between the original attempt and the retry, so neither should be used. A common reference architecture formalizes this as keys derived from the pair (workflow_id, step_id), which is a useful way to think about it: the key comes from the position in the workflow, not from anything generated in the moment. A practical version of this is a hash of run_id plus step_index plus action_type, which yields a key that's stable across restarts and doesn't depend on the model's output.

On the wire, this usually takes the form of an API header, Idempotency-Key being the common convention, and the receiving server checks its records for a request that already carries that key. If it has, it returns the cached result instead of executing again. Stripe supports this natively, and other payment APIs offer similar support, though the deduplication window varies by provider: Stripe's is 24 hours, and others differ meaningfully from that, so the window itself has to be a design decision, not an assumption.

For APIs that don't support idempotency keys at all, the burden shifts to the application layer: check a sent-log before sending, every time, no exceptions. The fourteen-email incident happened precisely because that check was absent.

Pattern 2: Deduplication tables and execution ledgers as the safety net for side effects the key alone cannot cover

An idempotency key tells the server what to compare against. It doesn't, by itself, give the orchestrator a record of what it has already done. That's what the deduplication table is for: a record keyed on the idempotency key, carrying a status field (pending, complete, or failed) and the cached result of the action once it completes. Every side-effecting action goes through this table. No exceptions, because the exceptions are exactly where the next incident comes from.

The check-then-act order matters more than it looks like it should. Before executing anything, the orchestrator checks the table first. If a completed record already exists for that key, it returns the cached result and never touches the tool again. If no record exists, only then does it execute. But the insert has to happen before execution, not after: claim the slot first, execute second. If the sequence runs execute-then-log and the process crashes in between, the next retry sees no record at all and re-executes the side effect from scratch, which defeats the entire point of the table.

For high-cost or financial actions, a second layer of logging earns its cost: log intent before executing. "Agent is about to transfer £5,000 from account A to account B with idempotency key XYZ," written before the transfer call goes out. If the agent crashes between that log and the transfer completing, the log on restart shows intent without a completion record, and the orchestrator can make a deliberate decision, retry or roll back, instead of guessing.

None of this, on its own, covers concurrent agents. The sequence of marking a record in-progress and then marking it complete is not atomic across separate processes, so two agents acting on the same session ID at the same time can both pass the check-table step before either one writes its claim. Covering that case needs a distributed lock, a Redis SET NX call or a database row lock being the standard mechanisms, layered on top of the dedup table rather than instead of it.

Pattern 3: Saga compensation for multi-step workflows where a later step fails after earlier steps have already committed

Single-tool idempotency is the tractable half of this problem. When an agent calls several tools in sequence and something fails partway through, retrying the whole sequence at that point doesn't recover from the failure, it compounds whatever already committed.

The saga pattern handles this by giving every step a compensating action, one that reverses the step's effect if something later in the chain fails permanently. It's eventual consistency achieved through explicit reversal, not through atomic rollback, since there usually isn't a single transaction spanning all the steps to roll back. In a hotel booking sequence, reserving inventory pairs with releasing the reservation, charging payment pairs with issuing a refund, and sending a confirmation pairs with sending a cancellation notice. If the confirmation step fails permanently after the first two steps have already committed, the saga executor runs the compensating actions in reverse order, unwinding the charge and then the reservation.

Compensating actions need the same idempotency discipline as the original calls. A refund that fires twice because the compensation step itself got retried is the identical class of bug as a charge that fires twice, just running in the opposite direction.

The dedup table from Pattern 2 is what makes saga state readable on restart. The orchestrator checks which steps carry a status of complete, which are still pending, and which failed outright; steps that already completed can be skipped on resume, while failed ones trigger compensation on the steps that already ran. Without that table, there's no reliable way to know where in the saga the failure actually occurred.

Pattern 4: Retry budgets and tool-level error contracts that prevent the model from retrying its way into a deeper hole

A retry loop with no ceiling on cost is a reliability failure in its own right, independent of everything above. The naive version of the agent loop retries as many times as it takes, which turns a transient, recoverable failure into a duplicate-action cascade the moment the underlying cause isn't actually transient.

The fix is a retry budget rather than a retry loop: a fixed maximum number of attempts, tracked at the orchestrator level rather than buried inside the tool call itself. Once that budget runs out, the correct behavior is to surface a structured failure and stop, not to keep trying with no plan for what happens if it never succeeds.

Some failures don't deserve a retry. A card decline, a validation error, a 400 Bad Request: none of these change on a second attempt, so retrying them wastes the budget and risks duplicating whatever side effect did land. A 503 or a network timeout is a legitimate retry candidate, because the failure there is plausibly transient. The tool's error response has to say which kind of failure occurred, explicitly, with a field like retryable: true or false and, where relevant, a retry_after_seconds value. Absent that signal, the orchestrator has no way to tell the two failure classes apart and ends up applying the same retry logic to both, which is exactly the collapse that produced the double subscription charge.

Errors, in this model, are tool results, not exceptions. Propagating a structured error object back to the model, rather than raising an exception that skips past the retry-budget logic entirely, matters because the model reasoning over a structured error that captures what failed, its retryability, and the state the system was left in can make a better call than an orchestrator trying to infer intent from a raw stack trace after the fact.