Automatic prefix caching helps agent workloads when many requests begin with exactly the same token sequence: a long system policy, tool catalog, repository context, document, or earlier conversation. Put stable content first, serialize it deterministically, keep request-specific data after it, and isolate cache reuse with a per-tenant or per-trust-group salt. Then prove the benefit with cached-token and prefill-latency measurements; a long prompt that changes near the beginning will not reuse much, no matter how similar it looks to a person.
vLLM's V1 design hashes full KV-cache blocks from their token IDs, parent-prefix hash, and relevant extra values. A later request can reuse matching blocks rather than recompute their key/value tensors. Only full blocks are cached, and the matching unit is tokenized input—not raw text or a semantic similarity score (vLLM Automatic Prefix Caching design).
Know what APC improves
Transformer inference has a prefill phase that processes prompt tokens and a decode phase that generates output tokens autoregressively. Reusing a prompt prefix reduces prefill work for the matched blocks. It can improve time to first token and free compute for other requests, especially when the shared prefix is long.
It does not eliminate:
- prefill for the unmatched suffix;
- decoding the response;
- tool execution or network latency;
- tokenization and request handling;
- cache lookup and scheduling overhead; or
- work after the cached blocks have been evicted.
Short prompts, mostly unique contexts, high output-to-input ratios, frequent restarts, or a cache under heavy pressure may show little benefit. APC also consumes the same finite KV-block pool used by active requests; cached blocks with no active reference are eviction candidates, not permanently reserved memory.
Make prefixes identical by construction
An agent prompt often contains the right reusable material in the wrong order:
request timestamp changes every call
user identity changes every call
user question changes every call
system policy stable
tool definitions stable for a release
large reference document shared for a task group
Reorder the rendered prompt when the model's template and security design permit:
stable chat-template framing
stable system policy
stable, deterministically ordered tool definitions
shared document or repository snapshot
conversation prefix
request-specific user message, timestamp, nonce, and dynamic state
Never reorder roles or fields contrary to the model's documented chat template. The goal is to make application inputs stable before the tokenizer, not manually splice token IDs into a chat protocol you do not control.
Canonicalize tool schemas and other maps using one serializer. Sort tools by a stable identifier if tool order has no intended meaning; preserve JSON array order where it does. Eliminate volatile request IDs, timestamps, trace headers, and whitespace from the stable region. Pin tokenizer and template revisions: a seemingly harmless template change can alter all downstream tokens and invalidate reuse.
Use cache boundaries that match product boundaries
Good reusable units include:
- a versioned global system prefix that contains no tenant data;
- a tenant policy and tool catalog shared only inside that tenant;
- an immutable document snapshot shared by users authorized for it;
- a repository commit shared by agents working on that commit; and
- the common beginning of a multi-turn conversation.
Do not concatenate unrelated customers' sensitive context merely to produce a popular prefix. Cache matching is an optimization, not authorization. The request must pass normal access checks before protected context is rendered.
Version stable content explicitly in your application manifest, but avoid inserting a version label before the actual shared content unless it is necessary: any changed early token ends reuse from that point forward. When a policy changes, changing its tokenized content naturally prevents reuse of blocks beyond the divergence.
Enable and pin vLLM behavior
The current vLLM V1 cache configuration reports prefix caching enabled by default, but defaults can change. Make the choice explicit in deployment configuration and record the runtime version:
vllm serve <model-id> \
--enable-prefix-caching \
--prefix-caching-hash-algo sha256 \
--generation-config vllm
Confirm the flags against the installed version. As of the freshness date, vLLM documents sha256 as its default hash algorithm. It serializes some hash inputs with Python pickle and may not reproduce hashes across Python or vLLM versions. sha256_cbor uses canonical CBOR for reproducible, cross-language hashing. Noncryptographic xxHash variants trade collision resistance for hash speed; vLLM warns that collisions can have undefined behavior or leak private information in a multi-tenant setting. Prefer the cryptographic option unless a measured, reviewed need justifies otherwise.
The hash also accounts for extra inputs such as LoRA identifiers and multimodal hashes, so visually identical placeholder tokens for different images should not share the wrong representation. Treat this as runtime behavior to regression-test when upgrading.
Isolate tenants with cache_salt
Shared-cache timing can reveal whether another request populated a prefix: a hit can be faster than a miss. vLLM supports a request cache_salt that participates in the first block's hash, so only requests with the same salt can reuse the chain.
Set the salt in a trusted gateway after authentication:
request["cache_salt"] = derive_opaque_cache_domain(
tenant_id=actor.tenant_id,
policy_epoch=current_policy_epoch,
)
Do not let the model, document, or unauthenticated client select another tenant's salt. Use an opaque, unguessable mapping if knowledge of a tenant ID would enable probing. Rotate the domain when access boundaries change, while recognizing that rotation sacrifices existing hits. A global salt is appropriate only for content explicitly safe to share globally; one salt per request prevents useful cross-request reuse.
Salting limits reuse, but it does not encrypt KV memory, authorize prompts, clear memory at a guaranteed time, or protect a compromised inference process. Keep process, GPU, cache directory, metrics, and administrative access isolated according to the data classification.
Preserve reuse across an agent loop
After each tool call, append messages rather than re-rendering earlier content in a different format:
turn 1: [system][tools][user]
turn 2: [system][tools][user][assistant tool call][tool result]
turn 3: [system][tools][user][assistant tool call][tool result][assistant]
The later request can match the full earlier prefix up to the new suffix if the serialization is byte-for-byte stable before tokenization. Common cache killers include:
- regenerating tool-call IDs inside the historical messages;
- changing tool order between turns;
- inserting the current time into the system prompt;
- summarizing or rewriting old turns on every step;
- switching chat templates or tokenizer revisions; and
- normalizing Unicode or whitespace inconsistently.
Conversation compaction intentionally changes the prefix. Measure its token savings against the lost cache hit instead of assuming one always wins.
Benchmark cold, warm, and pressured caches
Create paired cases with an identical prefix and controlled suffixes:
- Restart the server or otherwise establish a documented cold state.
- Send a priming request and wait for completion.
- Send a different request with the same tokenized prefix and salt.
- Repeat with one early-prefix token changed as a miss control.
- Repeat under target concurrency and realistic prompt/output distributions.
- Add enough distinct prefixes to exercise eviction.
Keep output lengths and sampling settings comparable. Record:
- total prompt tokens and cached prompt tokens;
- newly computed prefill tokens;
- prefix cache query/hit tokens and hit ratio;
- request prefill time and TTFT;
- end-to-end latency and TPOT;
- KV-cache usage, preemptions, queue time, and throughput; and
- process restarts, model/LoRA identity, salt domain, and request ordering.
Current vLLM production metrics include vllm:prefix_cache_hits, vllm:prefix_cache_queries, vllm:prompt_tokens_cached, per-request computed prefill tokens, prefill time, TTFT, queue time, and KV-cache usage (vLLM Production Metrics). Metric names have a documented deprecation lifecycle, so pin dashboards to your runtime version.
Do not infer hits from latency alone. Queueing, GPU clocks, batching, and compilation can make a miss faster than a busy hit. Use engine counters and paired latency evidence.
Failure modes and safeguards
Low hit rate: inspect tokenized prefixes, not rendered text. Diff token IDs and locate the first mismatch.
Hits disappear under load: the shared blocks may be evicted. Measure idle-before-eviction and cache pressure; reduce prefix diversity or adjust capacity only after confirming the cause.
No TTFT improvement: unmatched suffix, queueing, tokenization, or output work may dominate. Break latency into phases.
Cross-tenant timing signal: enforce gateway-derived salts and test two tenants with identical protected prompts.
Stale content concern: APC reuses computation for exact tokens; it does not fetch mutable source data. Resolve the current source revision before rendering and include changed content when the source changes.
Hash/version mismatch: a rolling deployment can have different local caches. Treat hits as opportunistic and never require them for correctness.
Production checklist
- Put stable, shareable content before dynamic request data.
- Canonicalize tools and templates, and pin tokenizer revisions.
- Make APC and hash algorithm explicit in deployment configuration.
- Derive cache salts from authenticated trust domains outside the model.
- Preserve historical message serialization across agent turns.
- Authorize source data before rendering it into a cacheable prefix.
- Measure cached tokens, computed prefill, TTFT, pressure, and eviction.
- Test cold, warm, early-mismatch, cross-tenant, and high-cardinality cases.
- Treat cache reuse as optional for correctness and availability.
- Revalidate security and hashing behavior after every runtime upgrade.