Backpressure is how an AI-agent system tells upstream callers that downstream capacity is full. A reliable design admits only work it can bound, limits queues and concurrency per scarce resource, propagates deadlines, and sheds or degrades excess load. Without those controls, agent fan-out and retries turn a slow dependency into a system-wide outage.
Why agents amplify overload
A conventional request may call one service. An agent request can create a plan, launch several subagents, query retrieval, call multiple tools, invoke a model again, and retry failures. The visible request rate is therefore not the resource arrival rate.
Model the amplification explicitly:
effective model calls = admitted tasks × model calls per task × attempts per call
effective tool calls = admitted tasks × tool branches per task × attempts per branch
Both multipliers are distributions, not constants. A rare task that recursively fans out can dominate a queue. Record child count, attempts, token reservations, and tool invocations per root workflow.
Google’s SRE analysis of cascading failures explains the dangerous feedback loop: overload raises latency, queues retain more work and memory, timeouts trigger retries, and retries add load. The cure begins at admission, before expensive work starts.
Put admission control at the front door
Admission control decides whether the system can accept a unit of work under its declared deadline, cost budget, tenant policy, and resource needs. Return a typed outcome:
accepted: capacity and budgets reserved;deferred: placed in a durable asynchronous class;degraded: accepted with a documented lower-cost plan;rejected_overload: retry guidance may be supplied;rejected_policy: retrying unchanged will not help.
Do not accept everything and call a full queue “backpressure.” By then the caller has no useful choice and the system has retained the cost of each request. Google SRE’s overload guidance recommends load shedding and graceful degradation so healthy capacity remains available.
The model must not decide whether its own work bypasses admission. Resource class, tenant priority, maximum fan-out, and budget come from trusted policy code.
Bound every queue
An unbounded queue converts a capacity problem into unpredictable latency and memory exhaustion. For each queue, define:
| Control | Required decision |
|---|---|
| Maximum depth | how much memory or durable storage is safe |
| Maximum age | when work can no longer meet its objective |
| Work class | interactive, batch, repair, evaluation, or other |
| Fairness | per-tenant share and anti-starvation rule |
| Overflow | reject, defer, or degrade |
| Cancellation | how queued work is removed and reservations released |
Queue age is more actionable than depth alone: ten slow, expensive workflows can be worse than hundreds of tiny tasks. Estimate work with tokens, expected tool units, or historical cost by task class. Correct estimates after completion, and cap the damage from underestimation.
Separate interactive and batch queues. A bulk evaluation should not consume every slot needed for user-facing work. Reserve capacity deliberately; do not rely on whichever request arrives first.
Limit concurrency at each bottleneck
One global semaphore cannot protect heterogeneous dependencies. Use independent bulkheads for model providers, model families, retrieval, browser tools, code sandboxes, databases, and tenant classes. A provider quota and a database connection pool fail at different thresholds.
type Permit = {
release(): void;
};
async function runWithPermit<T>(
limiter: Limiter,
deadlineMs: number,
operation: (signal: AbortSignal) => Promise<T>,
): Promise<T> {
const permit = await limiter.acquire({deadlineMs});
try {
return await operation(signalForDeadline(deadlineMs));
} finally {
permit.release();
}
}
Acquisition must honor cancellation and deadlines. Release the permit in every success and failure path. A leaked permit creates a slow outage; a released-too-early permit makes the limit fictional.
Our bounded-parallelism guide shows how to size and cancel a worker pool. Backpressure adds the upstream contract: what happens when no slot can be obtained in time.
Propagate capacity signals upstream
For synchronous HTTP, use explicit overload status and bounded retry guidance. For streaming transports, stop accepting new work when the receiver’s window is exhausted. For message systems, reduce consumption or stop acknowledging new messages while respecting lease and redelivery semantics.
Internal APIs should return structured reasons such as provider_concurrency, tenant_budget, or queue_deadline, not a generic exception. Upstream can then choose a cheaper route, defer, or fail without guessing.
Do not expose internal queue details that enable quota probing. External error messages can remain coarse while internal telemetry records the protected reason code.
Make deadlines consume the same finite budget
Set one absolute deadline at admission and propagate the remaining time to every child. Refuse to start work whose worst-case cleanup would exceed that time. Cancel children when the parent ends. The latency-budget guide covers critical-path measurement in detail.
Avoid a full timeout at every layer. A root with 30 seconds cannot safely launch three sequential tools that each believe they have 30 seconds. Queue wait, retries, and cleanup all consume the root budget.
Durable asynchronous workflows still need deadlines: expiration, maximum attempts, and a terminal state. Use the state-machine and idempotency patterns in durable AI agents so redelivery does not duplicate side effects.
Control retry amplification
Retry only errors classified as transient and only at one designated layer. Apply exponential backoff with jitter, cap attempts, and consume a retry budget shared by the root workflow. Honor provider retry-after guidance when compatible with the root deadline.
SRE guidance warns that retries at multiple layers multiply. If three layers each attempt a call three times, one root operation can create up to 27 downstream attempts before agent fan-out is counted.
Circuit breakers can stop repeated calls to a failing dependency, but they are not a queue. Define how probes occur and what degraded behavior is safe. A breaker that sends all work to one fallback can simply move the overload.
Degrade in ways that preserve correctness
Predefine degradation ladders by task class:
- Reduce optional retrieval breadth or reviewer branches.
- Use a cached immutable artifact after reauthorization.
- Switch to an approved lower-cost model only if its quality floor is established.
- Return a partial result with explicit limitations.
- Convert to asynchronous completion after an explicit user choice.
- Reject before side effects begin.
Never silently remove a safety check, authorization step, or required validation. Graceful degradation means less optional work, not weaker trust boundaries. If no safe reduced mode exists, fail closed.
Prevent noisy-neighbor failures
Apply per-tenant concurrency, rate, token, and cost limits in addition to global limits. Use weighted fair scheduling or reserved pools for workload classes. Measure both offered and admitted load so a healthy success rate does not hide that one tenant is being rejected disproportionately.
Priority must come from authenticated policy attributes. Do not accept a prompt that says “this is urgent” as a scheduler override. Cap high-priority capacity too; otherwise a flood of privileged work can starve recovery operations.
Prove recovery under load
Test more than the nominal maximum. Run:
- steady arrival below capacity;
- a ramp through the admission threshold;
- a sudden burst;
- a slow or rate-limited model provider;
- a tool outage during fan-out;
- cancellation of queued and running work;
- recovery after dependency latency normalizes.
An open-loop arrival test reveals whether queues grow when callers do not wait for replies. A closed-loop concurrency test reveals behavior for a fixed number of active users. Use both. Verify that the system rejects quickly, preserves protected workloads, stops retry amplification, releases permits, drains without another spike, and returns to normal automatically.
Track admitted/rejected/deferred rates, queue age, in-flight units, permit wait, deadline remaining at dispatch, fan-out, retry volume, cancellations, and resource saturation. Pair them with end-to-end success and quality; an empty queue achieved by dropping valid work is not success.