Durable AI Agents: Retries, Idempotency, Human Approval, and Resume

A durable agent is not an endless chat loop. It is a state machine whose decisions, side effects, approvals, and recovery rules remain correct when processes crash or messages arrive twice.

The central problem is ambiguity. If a payment API times out, did the charge fail, or did the response get lost after the charge succeeded? If a human approves a deployment, did they approve the current plan or an older one? A reliable design records enough state to answer those questions without asking a model to guess.

Split deterministic orchestration from fallible work

Use two layers:

• Workflow/orchestration: deterministic state transitions, timers, retry policy, cancellation, and approval state.
• Activities/tools: model calls, network requests, database access, emails, payments, and other failure-prone operations.

The workflow decides what should happen. Activities interact with the outside world. This is the same separation used by durable execution systems such as Temporal: its retry documentation directs nondeterministic operations such as API and LLM calls into Activities, while Workflow code must remain deterministic for replay.

Define the state machine first

For an agent that prepares and executes a consequential action:

received
  → planning
  → awaiting_approval
      ├── rejected → cancelled
      └── approved → executing
                        ├── succeeded
                        ├── failed
                        └── outcome_unknown → reconciling

Every state has allowed events. A late approve event cannot move a cancelled run back to executing. A second execute command cannot create another side effect after success.

A framework-neutral TypeScript model makes those rules reviewable:

type RunStatus =
  | "received"
  | "planning"
  | "awaiting_approval"
  | "executing"
  | "reconciling"
  | "succeeded"
  | "failed"
  | "cancelled";

interface ProposedAction {
  readonly actionHash: string;
  readonly summary: string;
  readonly tool: string;
  readonly arguments: unknown;
}

interface RunState {
  readonly id: string;
  readonly status: RunStatus;
  readonly proposal?: ProposedAction;
  readonly version: number;
}

actionHash is a digest of a canonical representation of the exact tool, arguments, target, and relevant policy version. Approval references that digest. If the model changes any consequential field, the hash changes and the workflow returns to awaiting_approval.

Persist state and history separately

Store a current snapshot for efficient reads and an event history for audit and recovery:

CREATE TABLE agent_run (
  id uuid PRIMARY KEY,
  tenant_id uuid NOT NULL,
  status text NOT NULL CHECK (status IN (
    'received', 'planning', 'awaiting_approval', 'executing',
    'reconciling', 'succeeded', 'failed', 'cancelled'
  )),
  state jsonb NOT NULL,
  version bigint NOT NULL DEFAULT 0,
  created_at timestamptz NOT NULL DEFAULT now(),
  updated_at timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE agent_run_event (
  run_id uuid NOT NULL REFERENCES agent_run(id),
  sequence bigint NOT NULL,
  event_type text NOT NULL,
  payload jsonb NOT NULL,
  created_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (run_id, sequence)
);

Update the snapshot and append the event in one database transaction. Use optimistic concurrency on version so two workers cannot both advance the same run:

UPDATE agent_run
SET state = $1, status = $2, version = version + 1, updated_at = now()
WHERE id = $3 AND version = $4;

Exactly one row must change. If zero rows change, reload and decide whether the event is a duplicate, stale, or invalid for the new state.

Apply the same tenant authorization or row-level-security boundary to run, event, and effect tables; a foreign key establishes relationship, not permission.

Do not store hidden chain-of-thought. Persist observable decisions, model and prompt versions, tool proposals, approvals, results, errors, and source references.

Make every side effect idempotent

Give each logical activity a stable identity derived from the run and workflow step:

idempotency_key = run_id + ":send-approved-refund:v1"

Persist the attempt before or atomically with execution where possible:

CREATE TABLE activity_effect (
  idempotency_key text PRIMARY KEY,
  run_id uuid NOT NULL REFERENCES agent_run(id),
  activity_type text NOT NULL,
  request_hash text NOT NULL,
  status text NOT NULL CHECK (
    status IN ('started', 'succeeded', 'failed', 'outcome_unknown')
  ),
  external_reference text,
  result jsonb,
  updated_at timestamptz NOT NULL DEFAULT now()
);

On retry:

1. look up the idempotency key;
2. verify the request hash is identical;
3. return the stored result if it succeeded;
4. reject key reuse with different arguments;
5. reconcile started or outcome_unknown with the external system; and
6. execute only when policy says no prior effect exists.

Prefer downstream APIs that accept their own idempotency key. Store the downstream operation ID so a status endpoint can resolve ambiguous timeouts.

If an API offers neither idempotency nor lookup, automatic retries may be unsafe. Route the run to human reconciliation.

Retry by failure class

Retries help transient failures and multiply permanent failures.

Set an overall schedule-to-close or workflow deadline. A maximum attempt count alone can still wait too long; an unlimited retry policy can become a denial-of-wallet loop.

Temporal Activities retry by default with exponential backoff, while Workflows do not retry by default. Its documentation also recommends marking known permanent failures as non-retryable rather than burning attempts on unchanged bad input.

Make approval a durable event

A safe approval record contains:

• run and tenant identity;
• approver identity and authorization source;
• exact action hash;
• human-readable preview shown to the approver;
• decision and optional reason;
• timestamp and expiry; and
• policy version used to require approval.

Approval must arrive through an authenticated application path, not as model-generated text. The workflow verifies that the approver is allowed to approve this action and that the proposal hash still matches.

If approval expires, the run remains paused or is cancelled according to policy. Do not let a model infer approval from silence or from text on a webpage.

Durable workflow engines expose message-passing primitives for this. Temporal, for example, supports Queries, Signals, and Updates: Queries read workflow state; Signals asynchronously change it; Updates can validate, change state, and return a result. Choose the primitive based on the acknowledgement semantics your approval UI needs.

Resume after a crash

On worker restart, the engine or recovery loop reloads non-terminal runs. For each run:

1. reconstruct state from the persisted snapshot/history;
2. inspect the current activity effect record;
3. reconcile any ambiguous external operation;
4. reschedule only commands that are safe to attempt; and
5. continue timers or remain paused for approval.

Never “resume” by replaying an entire transcript through the model and hoping it reaches the same decision. Model output is nondeterministic, tools may have changed, and old external content may now be unavailable.

If you use a replay-based engine, changing workflow code can break determinism. Follow its versioning/deployment mechanism and test replay against stored histories before deployment.

Handle cancellation and compensation explicitly

Cancellation is a request, not time travel. It can stop pending and cancellable work, but it cannot unsend an email or undo a settled payment.

For each activity, define:

• whether it observes cancellation;
• the last safe point to cancel;
• whether it has a compensating action;
• who may trigger compensation; and
• what happens if compensation also fails.

Compensation is a new side effect with its own idempotency key and audit trail. “Refund payment” is not a rollback of “capture payment”; it is another financial operation.

Protocol tasks are not the workflow database

MCP’s experimental Tasks utility can represent deferred request results. A2A Tasks represent stateful agent work. These are useful interoperability contracts, but you still need durable persistence, authorization, retries, and recovery behind them.

Map external task states to your internal workflow carefully. Do not expose internal event history or accept a caller’s task ID as authorization.

Test the failure boundaries

Automate scenarios that kill the worker:

• before a side effect;
• after sending the request but before storing the response;
• after storing success but before advancing workflow state;
• while waiting for approval;
• after approval but before execution; and
• during compensation.

Also deliver every event twice, out of order, and after it has expired. Assert there is at most one logical external effect and that invalid transitions are rejected.

Trace each active segment with OpenTelemetry, and put these cases into CI evals. Durability bugs are easiest to find before the workflow holds real money, messages, or infrastructure access.

Sources and freshness notes

This guide was checked on July 18, 2026 against Temporal’s official durable execution overview, Retry Policy reference, and TypeScript workflow message-passing guide, plus the MCP Tasks utility. The architecture is engine-neutral; confirm exact APIs and replay/versioning rules for the engine you deploy.