Event-Sourced AI Agents: Replay, Debug, and Resume Any Run

An event-sourced agent stores each accepted state change as an immutable, ordered event and derives current state through deterministic projection. Replay means applying recorded events again; it does not mean calling the model or external tools again.

This design makes decisions, approvals, side effects, and failure states inspectable. It also adds schema evolution, privacy, storage, and projection complexity. Use it when audit, recovery, temporal debugging, or several downstream consumers justify that cost.

Decide what belongs in the event history

Record observable facts and decisions:

• run accepted with a goal reference and policy version;
• plan artifact proposed and validated;
• model operation completed with model and prompt versions;
• tool invocation requested, started, succeeded, failed, or became uncertain;
• evidence or artifact stored with a digest;
• human clarification or approval recorded;
• budget reserved, committed, released, or exhausted;
• workflow state transitioned;
• cancellation requested and acknowledged.

Do not store hidden chain-of-thought. Do not put raw secrets, access tokens, or unnecessary personal data into an append-only history. Store references to protected content when retention or deletion requirements differ from the event ledger.

The event history is not the same as an observability trace. A trace helps diagnose a request and may be sampled or redacted. The domain event log is authoritative input to state reconstruction and must meet stricter ordering, validation, and retention rules. You can correlate them with a trace ID.

Define an event envelope

interface AgentEvent<TType extends string, TData> {
  readonly eventId: string;
  readonly runId: string;
  readonly tenantId: string;
  readonly sequence: string; // Base-10 bigint assigned by the event store.
  readonly type: TType;
  readonly schemaVersion: number;
  readonly occurredAt: string;
  readonly recordedAt: string;
  readonly actor: {
    readonly kind: "user" | "agent" | "service" | "operator";
    readonly id: string;
  };
  readonly causationEventId?: string;
  readonly correlationId: string;
  readonly data: TData;
}

occurredAt describes when the source says something happened; recordedAt describes when the event store accepted it. Use store-assigned sequence for ordering within a run. The example represents it as a decimal string so a database bigint is not silently rounded by JavaScript; parse or compare it with arbitrary-precision code. Wall-clock timestamps are not a safe total order when events arrive concurrently or clocks differ.

CloudEvents 1.0.2 is an optional interoperable envelope for events crossing service boundaries. Its core specification requires id, source, specversion, and type, and requires producers to keep the source plus id combination unique for distinct events. It does not define your agent's domain event names, sequence semantics, or authorization model. CloudEvents 1.0.2 core specification

Use CloudEvents when ecosystem interoperability helps; do not add it merely to rename equivalent internal fields.

Append events with optimistic concurrency

A relational baseline stores events and a projection:

CREATE TABLE agent_event (
  tenant_id uuid NOT NULL,
  run_id uuid NOT NULL,
  sequence bigint NOT NULL,
  event_id uuid NOT NULL,
  event_type text NOT NULL,
  schema_version integer NOT NULL,
  occurred_at timestamptz NOT NULL,
  recorded_at timestamptz NOT NULL DEFAULT now(),
  actor jsonb NOT NULL,
  causation_event_id uuid,
  correlation_id uuid NOT NULL,
  data jsonb NOT NULL,
  PRIMARY KEY (tenant_id, run_id, sequence),
  UNIQUE (tenant_id, event_id)
);

CREATE TABLE agent_run_projection (
  tenant_id uuid NOT NULL,
  run_id uuid NOT NULL,
  version bigint NOT NULL,
  status text NOT NULL,
  state jsonb NOT NULL,
  updated_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (tenant_id, run_id)
);

Every command includes the projection version it observed. In one database transaction:

1. lock or compare the current version;
2. authorize and validate the command;
3. assign the next sequence;
4. append one or more events;
5. apply them to the projection;
6. increment the version;
7. commit.

If the version changed, reload and decide whether the command is a duplicate, still valid, or stale. Never let two workers both approve transitions from the same observed state.

PostgreSQL's Serializable isolation can roll back a transaction with serialization_failure when concurrent reads and writes would produce an outcome inconsistent with serial execution. Applications using that level must handle the retry at the transaction boundary; serializable isolation does not make external tool calls safe to repeat. PostgreSQL 18 transaction documentation

Keep model and network operations outside the database transaction. Append an intent, commit, perform the external operation with idempotency, then append the observed outcome.

Make projection pure and exhaustive

The projector is an ordinary function:

type RunStatus =
  | "received"
  | "planning"
  | "awaiting_input"
  | "awaiting_approval"
  | "executing"
  | "reconciling"
  | "succeeded"
  | "failed"
  | "cancelled";

interface RunProjection {
  readonly status: RunStatus;
  readonly version: string;
  readonly planArtifactId?: string;
  readonly pendingActionDigest?: string;
  readonly terminalReason?: string;
}

function applyEvent(
  state: RunProjection,
  event: AgentEvent<string, unknown>,
): RunProjection {
  switch (event.type) {
    case "run.plan_validated":
      return {
        ...state,
        status: "executing",
        version: event.sequence,
      };
    case "run.approval_required":
      return {
        ...state,
        status: "awaiting_approval",
        version: event.sequence,
      };
    case "run.cancelled":
      return {
        ...state,
        status: "cancelled",
        version: event.sequence,
      };
    default:
      throw new Error("Unsupported event type or schema version");
  }
}

A real projector switches on both event type and schema version and validates data before applying it. Unknown events should stop a critical projection rather than being silently ignored.

Projection code must not read the current time, call a model, fetch a URL, or inspect mutable configuration. If a decision depends on a policy or price table, the event records the relevant version and accepted outcome.

Record model outputs as outcomes

Model inference is not deterministic replay. Store enough information to audit the call according to privacy policy:

• provider and returned model identifier;
• prompt/template and tool-schema versions;
• context manifest or protected content references;
• inference configuration;
• request and response IDs;
• validated artifact or output reference;
• token usage as estimated or provider-reported;
• validation and repair outcomes.

On replay, consume the recorded validated outcome. Reissuing the call is a new experiment or branch, not reconstruction of the original run.

Likewise, replay must never resend an email, charge a card, or redeploy software. External effects are represented by intent and outcome events, with reconciliation for ambiguity. The durable-agent guide covers idempotency and outcome-unknown states.

Model commands, events, and effects separately

These words should not be interchangeable:

A model generally proposes a command or artifact. Policy validates it. Only accepted state changes become events.

Resume from state, not from the last sentence

After a crash:

1. load the latest trusted snapshot if one exists;
2. replay subsequent events in sequence;
3. find nonterminal intents;
4. reconcile external operations whose outcomes are unknown;
5. re-establish timers and pending human interactions;
6. resume only transitions allowed from the reconstructed state.

Snapshots are caches. Include the last applied sequence and projector version; verify them before use. You must be able to rebuild from the retained event stream or explicitly document when compaction makes that impossible.

Evolve event schemas without rewriting history

Prefer adding a new event version and an upcaster:

function upcast(event: StoredEvent): CurrentEvent {
  if (event.type === "run.approval_required" && event.schemaVersion === 1) {
    return {
      ...event,
      schemaVersion: 2,
      data: {
        ...event.data,
        expiresAt: null,
      },
    };
  }

  return validateCurrentEvent(event);
}

An upcaster transforms old representation into current semantics; it must not invent a historical decision that never occurred. Test every retained event version against the current projector before deployment.

Do not rename old event meaning in place. Event names should describe facts, not implementation steps: ActionApproved is more durable than ApprovalModalButtonClicked.

Plan privacy and deletion before append-only storage

Immutability can conflict with correction, deletion, and retention obligations. Minimize event payloads and separate:

• stable operational facts required for integrity;
• sensitive content in an access-controlled store referenced by ID;
• telemetry with its own sampling and retention;
• cryptographic keys if crypto-shredding is part of a reviewed design.

Deleting referenced content can preserve ledger shape while making a detailed replay incomplete. Document that tradeoff honestly. Tenant isolation must apply to events, snapshots, artifacts, and every projection; a correlation ID is not an authorization boundary.

OWASP identifies sensitive-data exposure, memory poisoning, and insecure inter-agent communication as agent risks. Append-only storage does not make untrusted data safe; retain provenance and validate every item before it can influence a new command. OWASP AI Agent Security Cheat Sheet

Verify with replay tests

Maintain fixtures for every event version and assert:

• replay from zero equals replay from each valid snapshot;
• duplicate command delivery does not append duplicate effects;
• invalid transitions are rejected;
• unknown event versions stop projection;
• concurrent commands produce one accepted sequence;
• projection rebuild matches the online projection;
• cancelled and terminal runs cannot resume;
• old events still upcast after schema changes;
• redacted content fails safely rather than being guessed;
• reprocessing never calls external systems.

Also run a production-safe projection rebuild in shadow and compare it with the live view before changing projector code.

Event-sourcing checklist

• Record observable facts, not hidden reasoning.
• Order events with a store-assigned per-run sequence.
• Authorize and append through optimistic concurrency.
• Keep external calls outside database transactions.
• Make projections pure, exhaustive, and version-aware.
• Replay recorded model and tool outcomes; never re-execute them.
• Separate commands, events, effects, artifacts, and projections.
• Reconcile incomplete intents before resuming.
• Test upcasters against every retained event version.
• Design minimization, retention, correction, and deletion up front.