Debug a production AI-agent incident by containing user harm first, establishing incident roles, preserving privacy-safe evidence, and reconstructing one root workflow across models, tools, memory, and side effects. Do not start by changing prompts at random. Stabilize the system, identify the first invariant violation, verify recovery with canaries, and reconcile every ambiguous external action.
Declare the incident around impact
Trigger incident response on user-visible or safety-relevant symptoms: unauthorized writes, sensitive-data exposure, materially wrong outcomes, stuck workflows, duplicate actions, widespread latency, runaway spend, or an unknown external state. A provider’s error rate is evidence, not the incident definition.
The Google SRE incident-management guide emphasizes preparation, clear roles, a managed response process, communication, mitigation, and learning. Assign at least:
- incident commander: owns priorities and decisions;
- operations lead: executes mitigation and investigation;
- communications lead: maintains stakeholder updates;
- scribe: records timeline, evidence, hypotheses, and commands.
One person may cover multiple roles in a small event, but name the ownership. Keep a timestamped log. Separate facts, hypotheses, decisions, and results.
Contain harm before seeking the perfect cause
Choose the smallest reversible control that stops impact:
- Disable the affected write tool or credential.
- Pause the workflow class or deployment.
- Route to an evaluated fallback or human queue.
- Reduce fan-out and concurrency.
- Stop automatic retries that may duplicate effects.
- Quarantine suspect memory, retrieval indexes, or cached artifacts.
- Revoke exposed secrets and isolate telemetry destinations.
If no safe degraded path exists, fail closed. Never remove authorization, policy checks, or review gates to restore throughput.
Record the exact control, operator, time, scope, and expected effect. A kill switch should live outside the agent’s decision loop; the model cannot be trusted to disable itself during a control failure.
Preserve evidence without spreading sensitive data
Snapshot deployment identifiers, prompt/template versions, model and API identifiers, tool schemas, feature flags, policy versions, routing configuration, data/index snapshots, and relevant provider request IDs. Preserve audit logs and external resource state.
Do not paste full customer prompts or credentials into an incident channel. Use least-privilege storage, redact exports, and record who accessed sensitive evidence. OpenTelemetry’s sensitive-data guidance recommends minimization and filtering; hashing predictable secrets is not necessarily anonymization.
If deletion or credential rotation is urgent, preserve only the evidence allowed by the deployment's documented privacy, retention, and incident-response rules. Confirm any applicable contractual or jurisdiction-specific preservation requirements; incident convenience is not a reason to ignore them.
Reconstruct one workflow end to end
Start with a single affected logical task, not aggregate graphs. Build a timeline:
admitted -> queued -> plan -> model response -> tool dispatch
-> tool acknowledgement -> retry/resume -> final user response
Join root workflow ID, execution and attempt IDs, trace ID, provider request IDs, tool idempotency key, and external resource ID. Compare:
- what the user asked;
- what trusted policy allowed;
- what context the model received;
- what structured tool arguments were validated;
- what the tool accepted;
- what external state actually changed;
- what the agent told the user.
The Google SRE monitoring workbook distinguishes user-impact indicators from diagnostic dashboards. Use the service-level symptom to bound the incident, then traces, logs, metrics, and audit state to explain it.
Find the first broken invariant
Classify the failure, but do not stop at the label:
| Area | Useful invariant |
|---|---|
| Admission | work starts only with valid tenant, deadline, and budget |
| Routing | only eligible, versioned models are selected |
| Prompt/context | trusted instructions are separated from untrusted data |
| Retrieval/memory | every item has provenance and current authorization |
| Tooling | arguments are schema-validated and policy-authorized |
| Side effects | writes use idempotency and produce auditable receipts |
| Orchestration | retries and resume cannot repeat completed effects |
| Output | success is claimed only after independent verification |
Ask where the invariant first became false. A hallucinated confirmation may be downstream of a tool timeout; a tool timeout may be downstream of overload; overload may be amplified by retries. Fixing only the final message leaves the causal defect.
Compare change dimensions systematically
Build a before/after matrix around the first affected event:
- application and orchestrator release;
- prompt and tool-description revision;
- model or provider route;
- safety and authorization policy;
- retrieval corpus or embedding version;
- memory write and cache namespace;
- tool/service deployment;
- feature flags and tenant configuration;
- traffic mix, queueing, quota, and provider status.
Correlating with a change is not proof. Roll back one reversible dimension when it is a plausible containment step, and observe a predeclared indicator. Preserve the failed configuration for later replay.
Provider behavior can change without your deploy. Save returned model identifiers and provider status, but avoid attributing cause until a controlled comparison or provider evidence supports it.
Treat ambiguous side effects as a separate workstream
A timeout after dispatch does not prove a write failed. Mark the outcome unknown, stop blind retries, and query the external system by idempotency key or business identifier. Reconcile each task into committed, not committed, duplicated, or still unknown.
For duplicates, define compensating action with the owning business team; do not let the agent invent a reversal. Notify affected users when policy requires it. Our durable-agent guide covers explicit side-effect and recovery states.
Keep incident mitigation and reconciliation visible as separate tracks. Service recovery is not complete user recovery when tickets, messages, payments, or code changes remain wrong.
Reproduce safely
Replay the minimum failing input against a sandbox with writes disabled or replaced by faithful simulators. Pin the incident’s prompt, model route where still available, tool schemas, policy, data snapshot, and orchestrator build. Remove customer data or obtain appropriate authorization.
Because model outputs can vary, run multiple trials and distinguish deterministic orchestration defects from stochastic model behavior. Capture structured decisions and tool arguments. Do not execute copied generated commands against production.
Add fault injection at the suspected boundary: timeout after tool acceptance, partial stream, duplicate delivery, stale memory, provider rate limit, or cancellation during a write. A credible fix should preserve the invariant across the failure, not merely avoid one recorded output.
Recover with a canary and explicit exit criteria
Define recovery criteria before re-enabling traffic:
- user-impact indicator returns within objective;
- unsafe or duplicate effects stop;
- queues and retries drain to normal;
- canary tasks pass correctness and policy checks;
- external-state reconciliation is complete or owned with a deadline;
- telemetry coverage is sufficient to detect recurrence;
- rollback remains available.
Canary the fixed version on synthetic traffic, then a small authorized slice. Increase gradually while watching root-task success, unknown side effects, tool outcomes, latency, retry amplification, and cost. Do not use the same broken evaluator as the only recovery signal.
Google SRE’s cascading-failure guidance explains why recovery can trigger another surge as queues and retries resume. Keep admission controls active and drain deliberately.
Write a blameless, technically specific postmortem
The Google SRE postmortem guidance advocates learning-focused, blameless analysis. Include impact, detection, timeline, contributing conditions, first broken invariant, mitigation, recovery, evidence, and what made the incident larger or longer.
Actions should change systems, not ask people to “be more careful.” Examples include schema validation at the tool boundary, idempotency enforcement, prompt-version canaries, authorization on memory reads, bounded queues, provider-route pinning, redaction tests, or an external kill switch. Assign owners and due dates, then exercise the controls.
Also record where evidence was missing. If you could not connect a provider request to a tool write without prompt content, add privacy-safe identifiers rather than enabling blanket content capture.
A compact incident checklist
- Declare impact, severity, commander, operations, communications, and scribe.
- Stop harmful writes, retries, fan-out, or routes with reversible controls.
- Preserve versioned configuration and privacy-safe evidence.
- Trace one logical task through every attempt and external effect.
- Identify the first violated invariant.
- Reconcile unknown and duplicate side effects independently.
- Reproduce in a no-side-effect sandbox.
- Verify the fix under the triggering fault.
- Canary with explicit recovery criteria and rollback.
- Publish and track a blameless postmortem.