An AI-agent latency budget is an end-to-end deadline divided among admission, queueing, retrieval, model inference, tool calls, orchestration, and delivery to the user. Build it from a specific user journey, measure every boundary with one monotonic clock, and enforce remaining-time deadlines at each hop. A budget that exists only in a dashboard cannot prevent slow work from accumulating.
Define the user-visible clock first
“Agent latency” is ambiguous. A streaming assistant has at least two user-visible outcomes:
- Time to first useful output: from accepted user action to the first meaningful response, not merely an empty stream frame.
- Time to completed outcome: from accepted action to a final answer or a clearly communicated asynchronous handoff.
Write the journey and workload class next to the objective. “Interactive code explanation for authenticated users” and “overnight repository migration” should not share a latency target. The Google SRE guidance on service-level objectives recommends choosing indicators from user needs and separating workloads with different expectations.
Choose a percentile and window, not an average. A statement such as “99% of valid interactive requests produce useful output within the target over 28 days” is operationally testable. Define valid requests, cancellations, client disconnects, and policy-blocked requests before collecting data so teams cannot improve the number by changing the denominator.
Draw the critical path
Trace one request and label these boundaries:
client -> edge -> admission/queue -> retrieval -> model planning
-> tools (possibly parallel) -> model synthesis -> stream/client
Measure at least:
| Stage | Start | End | Why it becomes slow |
|---|---|---|---|
| Admission | request received | admitted/rejected | tenant limits, overload |
| Queue | admitted | worker starts | insufficient capacity, unfair scheduling |
| Retrieval | query starts | usable context ready | index or reranker tail latency |
| Model TTFT | provider request | first output token | provider queue, prompt size |
| Generation | first token | final token | output length, decoding rate |
| Tools | dispatch | usable result | external tails, retries, fan-out |
| Orchestration | between operations | next dispatch | serialization, state storage |
| Delivery | server write | client receives | buffering, network, rendering |
Use monotonic elapsed time within a process; wall clocks can jump. Across processes, use distributed traces and treat timestamp differences cautiously when clock synchronization is imperfect. The current OpenTelemetry GenAI schema includes a time-to-first-token attribute in seconds; follow the pinned definition from the GenAI semantic conventions repository.
For browser diagnosis, the W3C Server Timing specification defines a response-header mechanism for exposing server metrics to a user agent. It remains a Working Draft as of this article’s review date, so use it as a diagnostic interface, not as the only source of record.
Budget the deadline, not a sum of percentiles
Suppose an interactive target is D milliseconds. Reserve explicit client and network overhead, then allocate the server-side remainder based on trace data:
server budget = D - client allowance - network allowance
remaining at stage i = request deadline - monotonic now
Do not add each stage’s p95 and call the result the request p95. Percentiles are not additive: slow stages can occur on different requests. For parallel tool calls, the critical-path duration is approximately the maximum branch duration plus coordination, not the sum. Derive candidate allocations from complete traces, then replay or load-test the combined path.
A useful budget table includes three numbers for each stage:
- Allocation: the design allowance.
- Observed distribution: p50, p95, and p99 by workload class.
- Enforced timeout: no greater than the remaining request deadline.
This distinguishes a forecasting assumption from measured behavior and an actual control.
Make queueing visible and bounded
Queue delay is often omitted because no model or tool span has started. Record admission time, queue-enter time, and worker-start time. A growing queue converts overload into latency and memory pressure; the Google SRE discussion of cascading failures recommends failing early under overload, propagating deadlines, and avoiding queues that merely postpone failure.
Set a maximum queue depth or age per workload class. Reject, defer, or degrade work that cannot plausibly finish before its deadline. A response that immediately says “queued for asynchronous completion” may be more useful than holding an interactive connection until it times out.
Autoscaling does not replace admission control. New capacity may arrive after the queued requests’ deadlines, while retries add still more load.
Propagate one deadline through the graph
At admission, calculate an absolute deadline. Each model, retrieval, and tool call receives the remaining time minus cleanup margin. Cancel child work when the parent is cancelled or cannot meet the deadline. Our guide to durable AI agents explains how to make cancellation and resumption explicit state transitions.
Avoid independent full-duration timeouts. If an agent gives retrieval 10 seconds, the model 30 seconds, and each of three sequential tools 20 seconds under a 45-second user deadline, the configured numbers are mutually impossible.
Retries consume the same budget. Retry only transient failures, use backoff with jitter, and cap attempts by remaining time and a global retry budget. SRE guidance warns that retries at every layer multiply load; designate one layer as the retry owner.
Treat fan-out as a tail-latency multiplier
If a workflow waits for every branch, its outcome is governed by the slowest branch. Google’s original “The Tail at Scale” paper explains why rare component delays become common user-visible delays at large fan-out.
Control fan-out with:
- Bounded concurrency rather than unbounded task creation.
- A branch deadline derived from the parent.
- Early stopping once sufficient evidence exists.
- Partial-result semantics defined before failure.
- Hedging only for idempotent operations and only after measuring its added load.
For worker-pool mechanics, see bounded parallelism for agent swarms.
Separate perceived speed from completed work
Streaming can improve time to first useful output without reducing completion latency. Track both. Also record the time to first meaningful token: emitting whitespace, a status event, or a generic “working” phrase should not satisfy the user-facing indicator.
Status updates can still help when tools are slow. They should name the state honestly—“checking the deployment logs”—and remain accessible to screen readers. Never fabricate progress percentages when the workflow has no reliable completion estimate.
Prompt compression, caching, faster models, and parallel tools are candidate optimizations, not universal answers. Each changes quality, cost, or failure behavior. Validate the full outcome, not just a stage duration.
Run a reproducible budget exercise
Build a representative corpus of task classes and pin prompts, tool versions, model identifiers, and data snapshots. For every case:
- Run multiple trials to expose stochastic and provider variance.
- Save complete trace structure without sensitive content.
- Record model usage, tool outcome, quality result, and cancellation.
- Calculate stage and end-to-end distributions by task class.
- Identify the critical path for slow successful requests.
- Change one control, then repeat against the same corpus.
Test steady load, bursts, a slow tool, provider throttling, and cancellation. A budget proven only with one local user has not tested queueing. Do not publish benchmark results unless the workload, environment, trial count, exclusions, and uncertainty are disclosed.
Alert on user impact, then diagnose by stage
The primary alert should use the end-to-end service indicator. Stage dashboards explain why it is burning. Show queue age, admitted/rejected work, TTFT, completion latency, tool tails, retry volume, cancellation lag, output length, and quality guardrails together.
When a stage exceeds its allocation but end-to-end service remains healthy, investigate rather than automatically page. When the user-facing objective burns rapidly, page even if every component average looks normal. Averages hide the exact tail the budget was built to control.