Build a Simulated World for AI Agent Testing

A simulated world for agent testing is a stateful, resettable environment with the same contracts and policy boundaries as production, but no path to real customers or irreversible effects. Give each case an initial snapshot, allowed interfaces, injected failures, and executable terminal-state assertions. Do not confuse a conversational mock with a world: the test must retain and expose state.

Model the minimum world that determines correctness

Start from decisions and effects, not visual fidelity. A support agent may need:

users and authenticated principals
orders, payments, shipments, and policy versions
mailboxes and message delivery
calendar events and conflicts
tool schemas, authorization, rate limits, and errors
virtual time and scheduled work
an append-only audit log

Implement only behavior needed to distinguish safe success from failure. A fake payment gateway needs authorization, capture, idempotency, declines, timeouts, and unknown outcomes; it does not need the provider’s private internal architecture.

WebArena’s original contribution was a reproducible set of functional websites with realistic tasks and end-state evaluation. That design is useful evidence for stateful testbeds, but its websites and tasks do not automatically represent another product. Zhou et al., WebArena

Give every case a complete manifest

This illustrative manifest makes hidden dependencies explicit:

id: support-refund-policy-017
worldVersion: retail-world-v4
clock: 2026-07-18T16:00:00Z
principal: customer_104
fixture: fixtures/refund-017.json
task: "Refund the eligible item; do not alter the ineligible item."
faults:
  - at: first_call
    tool: payment.refund
    outcome: timeout_after_commit
assertions:
  - refund_count(item_eligible) == 1
  - refund_count(item_ineligible) == 0
  - no_cross_tenant_reads()
  - audit_log_contains("refund", "approved")
limits:
  modelTurns: 12
  toolCalls: 20

Pin fixture, world, policy, schema, and clock versions. Record seeds when simulation choices use pseudorandomness. A seed alone is insufficient if unordered collections, concurrency, or dependency versions can change execution.

Build through ports shared with production

Keep the agent runtime dependent on interfaces, then bind those interfaces to production or simulation implementations:

interface MailPort {
  search(query: string): Promise<readonly Message[]>;
  send(draft: Draft, idempotencyKey: string): Promise<SendResult>;
}

interface ClockPort {
  now(): Date;
  sleepUntil(instant: Date): Promise<void>;
}

interface World {
  readonly mail: MailPort;
  readonly clock: ClockPort;
  snapshot(): Promise<unknown>;
  reset(fixture: unknown): Promise<void>;
}

Share schemas, error codes, authentication, and authorization policy where practical. Do not import production credentials or endpoints into the simulator. Add an environment assertion at process startup and network-level egress denial so a configuration error cannot reach production.

Contract-test simulated and production adapters against the same interface cases. Simulation drift is a product defect: a test that passes only because the fake accepts an impossible request provides false confidence.

Make time controllable

Real sleeping makes tests slow and flaky. Use a virtual clock for deadlines, token expiry, calendar boundaries, scheduled delivery, and backoff:

class VirtualClock implements ClockPort {
  constructor(private current: Date) {}

  now(): Date {
    return new Date(this.current);
  }

  async sleepUntil(instant: Date): Promise<void> {
    if (instant > this.current) this.current = new Date(instant);
  }
}

The implementation is intentionally minimal. A real scheduler must define how simultaneous events are ordered, how cancellation works, and how concurrent waits resume. Test daylight-saving and locale behavior only if product logic depends on local civil time; store instants unambiguously.

Make failures scriptable and observable

Inject faults by semantic operation and occurrence, not arbitrary wall-clock milliseconds:

• timeout before acceptance;
• timeout after a write committed;
• rate limit with a retry time;
• malformed, stale, or partial tool result;
• authorization revoked mid-run;
• duplicate message delivery;
• conflicting concurrent update; and
• service recovery after a circuit breaker opens.

The agent should not know which fault is scheduled. The evaluator does. Preserve requests, policy decisions, effects, and state diffs as observable events.

Test unknown outcomes explicitly. When a refund times out after commit, a blind retry without an idempotency key can duplicate the effect. The correct behavior may be lookup, reconciliation, or escalation depending on the service contract.

Evaluate terminal state first

Assertions should query canonical state after the run:

requested outcome reached
forbidden records unchanged
permissions and tenant boundaries respected
at-most-once effects honored where promised
required audit evidence present
no pending work left unintentionally

Use trajectory assertions only for required policy order, such as approval before sending. Avoid requiring a golden click or tool sequence when several routes lead safely to the same state.

τ-bench evaluates tool-agent-user interactions by comparing database state to an annotated goal and reports repeated-trial reliability. Its design is a useful pattern for dynamic simulated interactions, not a guarantee that an LLM user simulator behaves like real users. Yao et al., τ-bench

Treat simulated users as test doubles

A scripted user is deterministic but covers limited conversational variation. A model-based user can vary language and react to the agent, but may leak goals, violate the persona, or share the same biases as the system being tested.

Constrain model-based users with hidden state, allowed disclosures, and a maximum turn count. Validate their behavior independently and retain transcripts. For critical cases, use deterministic branches or human-authored interaction trees. Never interpret performance against one simulator as a population estimate for humans.

Reset and isolate every run

Run each case in a new namespace, database transaction, container, or independently keyed fixture. Reset caches, queues, object storage, clocks, identity state, and rate-limit counters—not only the primary database. Generate run-scoped domains and recipients so email or webhook mistakes cannot escape.

Parallel tests must not share sequence counters or mutable fixtures. After each run, assert no external network connection occurred and no unowned resource remains.

Validate the simulator against reality safely

Compare contracts and sanitized production traces, not live effects. Periodically ask:

• Are input and output schemas still identical?
• Are errors and retry semantics represented?
• Are policy versions synchronized?
• Do latency and rate limits need a separate load-test model?
• Which production failures cannot the world express?

Keep realism claims narrow. A functional browser testbed may reproduce DOM interactions but not anti-bot systems, geographic routing, or every accessibility technology.

Know when a smaller fake is enough

Do not build a general world for a pure transformation with no tools or state. A table-driven unit test is clearer for parsing a date or validating a structured response. Use a contract stub when the question is whether one request is formed correctly. Add a stateful simulator only when history, concurrency, identity, failures, or side effects determine correctness. Maintainability is part of fidelity: an elaborate world nobody can update will quietly diverge from production.

Simulation checklist

• Model state and effects, not just dialogue.
• Pin initial fixture, world, policy, schema, and virtual clock.
• Share contracts while physically excluding production credentials and egress.
• Script pre-commit, post-commit, partial, duplicate, and recovery failures.
• Assert canonical end state and forbidden effects.
• Treat user simulators as fallible test doubles.
• Reset all stores, queues, caches, clocks, and identity state per run.
• Contract-test adapters and audit simulation drift.
• Use fresh real-world feedback to expand cases without replaying customer data unsafely.