Shadow Testing AI Agents in Production

Shadow testing sends an authorized copy of production-shaped input to a candidate agent, discards its user-facing response, and prevents every candidate side effect. It can reveal distribution and integration failures missed by offline datasets, but it cannot validate real write behavior or prove a release safe by itself.

Shadowing is not a canary. A canary serves some real traffic and may create real effects; a shadow must not influence users or shared production state.

Decide whether production data is necessary

Use offline replay or sanitized evaluation data when it answers the question. Shadowing adds privacy, security, cost, and operational risk. It is justified when the unknown depends on current production distributions, request shapes, retrieval behavior, or latency under realistic load.

Define the decision before collection:

Candidate: immutable agent configuration digest
Eligible traffic: authenticated support chats in the approved region
Exclusions: minors, regulated categories, opted-out users, attachments
Sampling: deterministic 2% by request ID
Duration and budget: predeclared
Metrics: coverage, hard-policy attempts, paired task-quality review, latency, cost
Exit: stop on privacy breach, effect attempt reaching a live adapter, or load budget

Obtain legal, privacy, and security review for the exact data path. A production request may contain personal data even when logs normally omit it.

Tee traffic through a one-way boundary

Google’s SRE workbook describes traffic teeing as copying requests to a test deployment while production serves the user and the copied response is discarded. It warns that shared state, including caches, can distort measurements or create interactions between deployments. Google SRE, Canarying Releases

Use this architecture:

eligible request
  ├─ production agent → normal response and authorized effects
  └─ privacy filter → shadow queue → candidate → isolated tools/sinks → evaluation store

The shadow path must never sit on production latency. Bound its queue, shed load, expire stale work, and give production priority. Use deterministic sampling so related analysis is reproducible, while honoring documented participation and exclusion choices and confirming any privacy requirements that apply to the data path.

Never copy credentials or session tokens. Replace them with a shadow principal whose capabilities exist only in the test environment.

Give the candidate realistic reads and fake writes

There is no perfect state strategy:

Select per question. For candidate tool-choice evaluation, recorded results fail when it requests a different tool. For latency comparison, a fast simulator says little about production dependencies.

Replace every write-capable adapter with an isolated implementation. Do not rely on the prompt to avoid writes or on a boolean inside the production adapter. Remove production network routes and credentials, and assert at infrastructure level that no connection to live write endpoints occurred.

External reads can also have effects: opening tracking URLs, consuming one-time tokens, marking messages read, or triggering metered APIs. Classify interfaces by effect, not HTTP method.

Compare paired outcomes without assuming production is correct

Pair candidate and production by a non-identifying run key and record configuration fingerprints. Compare:

• whether each completed the observable task;
• hard-policy attempts and broker decisions;
• tool errors, retries, and unsupported calls;
• source/citation support;
• latency, tokens, and external cost; and
• escalation or abstention.

Production behavior is a baseline, not ground truth. If the candidate disagrees, determine correctness using deterministic state checks, policy, qualified human review, or a calibrated grader. Blind reviewers to system identity and randomize answer order.

Analyze disagreement cases, not only average scores. A candidate may improve common summaries while failing a rare permission boundary. Report coverage and missingness: candidate timeouts and invalid outputs must not silently disappear from the denominator.

Account for state and timing bias

If production acts before the shadow runs, later reads may observe production’s effects. That contaminates a comparison. Record the logical input snapshot or tool results before effects where permitted, or label the test as post-state observation.

Queue delay changes time-sensitive answers, policy versions, inventory, and authorization. Record source event time, shadow start time, and tool observation time. Expire cases whose delay exceeds the predeclared window.

Caching can distort both performance and behavior. Separate caches and rate-limit accounting so doubled traffic does not warm production caches, exhaust quotas, or change retrieval rankings. Google’s SRE warning about shared cache state applies directly here. Google SRE traffic-teeing caveats

Protect user data and hostile content

Apply data minimization before the shadow queue. Redact fields the candidate does not need, enforce regional storage, encrypt payloads, limit access, and expire raw content. Make deletion propagate to derived shadow artifacts where required.

Treat copied content as untrusted. A production prompt injection must not gain a new route to credentials, broad tools, or evaluator infrastructure. Use sink-only egress, least privilege, call/depth/cost limits, and tenant-isolated memory. Never train on shadow data merely because it was collected for evaluation; that is a separate purpose and approval.

Monitor the shadow system itself

Track eligibility counts, sampling decisions, privacy-filter failures, queue delay, dropped and expired work, candidate completion, attempted effects, blocked egress, cost, and evaluator missingness. Alert on deviations from the declared sample and stop automatically if isolation fails. Shadow telemetry must be distinguishable from production telemetry so candidate errors do not page the wrong operator or contaminate service-level indicators. Before shutdown, drain or expire queued work deliberately and verify deletion and retention jobs completed.

Stage the rollout after shadowing

A responsible sequence is:

1. contract and unit tests;
2. offline agent evals and adversarial cases;
3. recorded replay;
4. limited shadow with all effects suppressed;
5. broader shadow after privacy and load review;
6. human-supervised or read-only canary; and
7. progressively expanded canary with rollback.

Shadow success does not test authorization against real writes, user reactions, or feedback loops. Canarying introduces actual exposure and therefore needs stricter monitoring and rollback. The Google SRE workbook defines canarying as exposing a subset of production traffic to a change to limit release risk. Google SRE canary guidance

Shadow-test checklist

• Prove that current production data is needed for the decision.
• Define eligibility, exclusions, sample, duration, budget, metrics, and stop rules.
• Filter data before an asynchronous, bounded shadow queue.
• Replace credentials and physically remove live write access.
• Classify read endpoints that still create effects.
• Choose and disclose the state strategy and its fidelity limits.
• Blind paired review and keep missing runs in the denominator.
• Separate caches, quotas, memory, and telemetry from production.
• Record timing contamination and queue delay.
• Treat shadow evidence as one stage, not release approval.