Disaggregate Prefill and Decode for Long-Context Agents

Disaggregated prefill/decode is worth testing when long prompts create bursty compute that harms decode latency and you need to scale time-to-first-token capacity independently from token-generation capacity. It is not a default speedup. Both pools generally load model weights, KV state must move between them, and the router, network, retries, and cache lifecycle become part of every request. Adopt it only if a fixed-resource benchmark shows more requests meeting your TTFT and TPOT objectives than a well-tuned colocated baseline.

The prefill phase processes prompt tokens in parallel and creates the attention key/value state; decode consumes that state while producing tokens serially. DistServe formalized placing these phases on different GPUs and optimizing “goodput”—completed requests that meet service-level objectives—rather than raw token throughput alone (DistServe paper). vLLM exposes KV-transfer connectors and example P/D deployments, but connector support and configuration remain version- and topology-specific (vLLM Disaggregated Serving examples).

Start with the interference you need to remove

In a colocated engine, prefill and decode share compute, memory bandwidth, scheduler capacity, and KV-cache blocks. A newly arrived long prompt can affect the inter-token latency of running decodes. Chunked prefill can limit that interference without a second pool; batching and admission control may also solve it.

Measure the baseline first:

• prompt and output token distributions by route;
• TTFT, TPOT, inter-token, queue, and end-to-end latency percentiles;
• prefill and decode service time;
• scheduler preemptions and KV pressure;
• arrival bursts and concurrency; and
• the fraction of requests meeting all relevant SLOs.

If decode latency is already stable, transfer and duplicated capacity are unlikely to help. If long prefills cause the tail, compare disaggregation against chunked-prefill and workload-aware routing, not an intentionally untuned server.

Model the data path

A minimal flow is:

client
  -> P/D-aware router
       -> prefill worker: tokenize/render, run prompt, produce KV blocks
       -> KV connector: publish/transfer block metadata and bytes
       -> decode worker: import KV, generate and stream output
  -> client

The router must keep one request identity across phases, select compatible workers, propagate cancellation and deadlines, and expose an unambiguous final state. Prefill success is not request success; a decode node may never import the state.

Both sides must agree on:

• model and weight revision;
• tokenizer, chat template, and rendered token IDs;
• model architecture and KV layout;
• tensor/pipeline/context parallel configuration;
• KV data type, block size, and scaling metadata;
• connector protocol and version; and
• request, block, and engine identifiers.

Fail closed on a mismatch. Importing KV produced for different tokens or weights is a correctness error, not a cache miss.

Estimate KV-transfer cost explicitly

For a conventional uniform decoder-only transformer, an unsharded planning estimate is:

KV_bytes_per_token = 2 * layers * kv_heads * head_dim * kv_element_bytes
transfer_bytes ≈ KV_bytes_per_token * transferred_prompt_tokens
transfer_time_lower_bound = transfer_bytes / measured_effective_bandwidth

The factor 2 is keys plus values. kv_heads must come from the model configuration; grouped-query and multi-query attention can use fewer KV heads than query heads. The actual transferred bytes depend on tensor/context sharding, page padding, hybrid or sliding-window layers, compression, connector framing, and whether cached blocks already exist at the decoder.

Measure the actual connector bytes and time. Link-rate marketing numbers are not application throughput, and transfer overlaps may make a simple sum inaccurate. A useful per-request decomposition is:

TTFT = router_queue
     + prefill_queue
     + prefill_compute
     + KV_publish_and_transfer
     + decode_queue_and_import
     + first_decode_step
     + network_streaming

Disaggregation wins only if reduced queueing/interference and better pool utilization outweigh the new terms under the workload that matters.

Size independent pools from demand

Prefill demand is driven primarily by uncached prompt tokens and their sequence lengths. Decode demand is driven by active sequences, output lengths, sampling/constraint work, and per-step batching. Do not size both pools from requests per second alone.

Use measured single-pool service curves:

prefill_required_token_service > arrival_rate * mean_uncached_prompt_tokens
decode_required_token_service  > arrival_rate * mean_generated_tokens

These are conservation checks, not capacity guarantees. Long-sequence attention cost is not necessarily linear in tokens, means hide bursts, and service rate changes with batch shape. Build histograms and simulate or replay the real arrivals. Add failure and maintenance headroom after measuring, rather than choosing a universal utilization target.

Each pool generally carries its own model weights and runtime overhead. Include those duplicated allocations in cost and available KV memory. A 1P/1D topology using two GPUs is not comparable with a one-GPU colocated server unless the decision is explicitly “buy another GPU.” For an efficiency comparison, hold total accelerator type and count constant.

Implement with a supported connector

vLLM's current examples include pull and push patterns and multiple KV connectors. The central configuration object is KVTransferConfig, but connector fields and operational guarantees differ. Start from the exact example shipped with your pinned release, vendor the configuration, and exercise it in a nonproduction network. The offline vLLM Disaggregated Prefill example is useful for understanding producer/consumer roles but is explicitly an example, not a production durability protocol.

Before adding a router, prove:

1. Prefill and decode produce the same token output as colocated inference for fixed fixtures.
2. A connector can transfer the maximum supported prompt without corruption.
3. KV blocks are released after success, cancellation, timeout, and worker crash.
4. Worker incompatibility is detected before import.
5. Internal transfer ports are unreachable from untrusted networks.

Then add routing, retries, and autoscaling. Never expose engine IDs, transfer descriptors, or connector control ports to clients.

Handle multi-turn agents deliberately

Agents repeatedly append tool results and continue a conversation. On the next turn, KV from the prior decode may reside only on a decode worker, while the new suffix requires prefill. Options include:

• recompute the conversation on a prefill worker;
• persist/cache KV in a connector-supported tier;
• transfer decode state back to prefill; or
• route sticky conversations through a topology that supports bidirectional reuse.

vLLM's example repository includes a multi-turn proxy showing bidirectional KV-transfer coordination, but examples evolve and do not define your retention or consistency policy. Measure the extra transfer and cache hit rate. A design optimized for one-shot long prompts can regress a multi-turn agent badly.

Conversation affinity must not override tenant isolation or overload one worker. On a miss, recomputation should remain correct. KV reuse is an optimization, not the source of conversation truth; retain canonical messages and tool artifacts in durable application storage.

Benchmark goodput, not an isolated phase

Compare at least:

1. colocated serving with tuned chunked-prefill/scheduling;
2. disaggregated serving with the same total hardware;
3. disaggregated pool ratios across a small predeclared grid; and
4. failure-degraded modes with a worker or transfer path unavailable.

Replay identical timestamped arrivals. Cover prompt/output length combinations, cache hits and misses, single- and multi-turn work, structured generation, and agent tool delays. Report:

• percent of requests meeting both TTFT and TPOT SLOs;
• TTFT, TPOT, ITL, queue, and end-to-end distributions;
• successful requests and tokens per second;
• per-pool queue depth, utilization, and KV pressure;
• bytes transferred, transfer/import latency, failures, and expiry;
• retries, duplicate work, orphaned blocks, and cancellations; and
• total GPU-hours and network use per successful request.

vLLM currently exposes NIXL connector counters for bytes, transfer time, failed transfers/notifications, expired requests, and descriptor counts alongside general latency and cache metrics (vLLM Production Metrics). Use the metrics for your selected connector; do not infer transfer success solely from the HTTP response.

Design failure semantics before production

Prefill worker fails before publication: retry safely on another compatible prefiller using the same canonical request.

Transfer outcome is unknown: identify the attempt uniquely and query/expire it; do not allow unbounded duplicate KV reservations.

Decode fails after import: retry only if sampling state, emitted tokens, and client streaming semantics can be reconstructed. Otherwise return an explicit interrupted state.

Client cancels: propagate cancellation to router, prefill, connector, and decoder; reclaim blocks with a bounded lease even if a message is lost.

Pool imbalance: apply backpressure and admission control. Scaling only decoders cannot fix a saturated prefill queue.

Version skew: drain incompatible workers; reject cross-version handoff rather than hoping the binary layout is stable.

Network exposure: vLLM's security guide says internal distributed and KV-transfer ports should be accessible only from trusted hosts or networks (vLLM Security). Authenticate and encrypt cross-host transport according to the connector and deployment environment.

Production checklist

• Prove prefill interference is the actual SLO problem.
• Benchmark chunked-prefill and routing controls as simpler baselines.
• Pin every model, KV-layout, parallelism, and connector parameter.
• Calculate and measure KV bytes, effective bandwidth, and transfer time.
• Hold total hardware constant for efficiency comparisons.
• Size P and D pools from prompt/output token demand and bursts.
• Test multi-turn KV ownership and retain canonical conversation state.
• Propagate deadlines, cancellation, identity, and tenant boundaries.
• Reclaim orphaned KV through leases and test every crash window.
• Gate rollout on fixed-resource goodput and total cost, not a prefill microbenchmark.