Caching AI Applications: Prompts, Prefixes, Retrieval, and Tools

Caching an AI application is safe only when the cache key represents every input that can change the result, the authorization check is repeated on every read, and invalidation follows the underlying data. Start with provider prefix caching, immutable retrieval artifacts, and idempotent tool reads. Cache final generated answers last because their correctness boundary is widest.

Choose the cache by what can be reused

“LLM cache” describes several different systems:

Do not put all four behind one generic cache.get(prompt). Their consistency, tenancy, cost, and failure policies are different.

Use provider prefix caching before inventing response caching

Prefix caching reuses computation for an identical leading sequence while the model still generates a new continuation. It can reduce latency and billed input work without pretending two user questions have the same answer.

Design the prompt from stable to variable:

tool definitions
stable system policy
large shared reference material
session context
user-specific request

The exact order and eligibility are provider-specific. Anthropic’s official prompt-caching documentation describes prefix matching across tools, system instructions, and messages, with default five-minute and optional one-hour cache lifetimes at the time of review. Google documents both implicit context caching and explicit cached-content objects, including usage metadata and explicit TTL management. Verify current model support and billing before implementation.

Self-hosted engines have their own boundaries. vLLM’s automatic prefix-caching design hashes full token blocks with their parent context and extra identifiers. Its security guidance recommends a cache_salt per trust group to isolate tenants and reduce timing inference. Treat that as implementation-specific guidance, not a universal API.

Measure hit eligibility, actual cache-read tokens, cache-write tokens, first-token latency, and output quality. A reordered tool definition or a timestamp embedded in the system prompt can destroy prefix hits even when the text looks substantially the same.

Build canonical, versioned keys

A cache key is a correctness proof. Construct it from canonical values, not string concatenation:

type RetrievalKey = {
  schemaVersion: 3;
  tenantScope: string;
  principalPolicyVersion: string;
  query: string;
  indexSnapshot: string;
  embeddingModel: string;
  filters: Record<string, string[]>;
  topK: number;
  rerankerVersion?: string;
};

Serialize with a specified canonical format, then compute a keyed HMAC for the stored key. A bare hash of low-entropy secrets can be guessed offline; OpenTelemetry’s sensitive-data guidance warns that hashing alone may not anonymize predictable values.

Include:

• tenant or trust scope;
• authorization/policy version;
• normalized inputs and locale;
• model and tokenizer identifiers where token identity matters;
• prompt/template and tool-schema versions;
• source snapshot or resource version;
• relevant decoding and safety settings;
• cache-key schema version.

Exclude incidental values only after proving they cannot affect the result. If order changes tool meaning, preserve order.

Cache retrieval artifacts, not assumed truth

For retrieval-augmented generation, cache document identifiers, scores, and source-version metadata rather than a final prose answer. On every hit:

1. Reauthorize the principal against each resource.
2. Verify the index or document version is still allowed.
3. Fetch current content or use an immutable snapshot.
4. Drop deleted or inaccessible results.
5. Re-run downstream synthesis under the current prompt and policy.

An ACL must never be “cached away.” Including a user ID in the key is insufficient if the user’s role changes after the entry is written. Version the authorization policy or perform a live authorization check.

Set TTL from the source’s acceptable staleness, not a convenient round number. A static product manual and an incident-status index need different policies. Prefer event-driven invalidation when the source emits reliable version changes; retain TTL as a backstop.

Cache tools only with explicit semantics

Classify each tool before caching:

• Pure read: same authorized input and resource version returns the same value. Usually cacheable.
• Time-sensitive read: cacheable only within a documented freshness window.
• Nondeterministic read: weather, market data, random sampling, or “current” queries require time/source versions.
• Write or side effect: do not replay from a result cache. Use idempotency keys to prevent duplicate execution instead.

The tool owns its cache contract. It should declare identity scope, freshness, version signal, negative-result policy, and whether errors are cacheable. Do not let an agent infer that a function called get_* is safe to cache.

Negative caching can protect a failing dependency, but distinguish “not found” from “temporarily unavailable” and keep failure TTLs short. Never cache an authentication failure as if the resource did not exist for every user.

Cache final responses only inside a narrow equivalence class

Exact final-response caching is justified when inputs are deterministic, fully represented, non-personal, and governed by an acceptable freshness window—for example, formatting an immutable public artifact under a pinned template. It is dangerous for personalized advice, live operations, permission-sensitive data, or tasks that require current tools.

Semantic caches widen the equivalence class by matching similar embeddings. Two semantically close questions can differ in jurisdiction, timeframe, negation, permissions, or required evidence. If you use one:

• restrict it to low-risk task classes;
• require exact matches on tenant, locale, policy, time, and source version;
• set a conservative similarity threshold from labeled data;
• store provenance and visibly refresh sources;
• fall back on uncertainty;
• audit false-hit rate by critical slice.

Do not use embedding similarity as authorization. OWASP’s sensitive-information disclosure guidance emphasizes preventing sensitive data exposure across model workflows.

Defend caches from prompt and memory poisoning

Cached retrieved text and tool results are still untrusted inputs. An attacker who causes malicious instructions to enter a shared cache can affect many later sessions. OWASP documents both prompt injection and persistent memory/context attack surfaces.

Controls should include:

• provenance and integrity metadata on every entry;
• tenant isolation and least-privilege cache access;
• length and type validation before writes;
• no executable interpretation of cached text;
• quarantine and purge by source/version;
• encryption in transit and at rest;
• sensitive-value suppression in keys, logs, and hit telemetry;
• a kill switch that bypasses a poisoned namespace.

Prefix-cache timing can reveal whether another request used the same prefix. Use provider isolation guarantees or per-trust-group salts; never assume an in-memory cache is harmless.

Prevent stampedes and silent fallback

When a popular entry expires, thousands of agents can recompute it simultaneously. Use request coalescing, bounded refresh concurrency, jittered TTLs, and stale-while-revalidate only where serving stale data is explicitly permitted. Keep hard maximum staleness.

Decide fail-open versus fail-closed per layer. A retrieval cache outage may safely fall back to live retrieval; an authorization-cache outage may need to fail closed. Record cache status as hit, miss, bypass, stale, or error. A cache error reported as a miss hides reliability incidents and can create a cost spike.

Evaluate correctness, latency, and cost together

Build a replay corpus containing normal requests plus permission changes, deleted documents, prompt-version changes, near-duplicate semantic queries, tenant boundaries, and cache outages. Compare cached and uncached runs on:

• task correctness and citation freshness;
• unauthorized or cross-tenant hit rate;
• exact and semantic false-hit rate;
• hit rate by layer;
• time to first token and completion latency;
• provider-reported cached-token usage;
• recomputation and storage cost;
• stampede amplification during expiry.

Pin model, prompt, data snapshot, and cache configuration. Run multiple trials for generated outputs. Do not claim savings from a synthetic hit rate alone; observed provider usage and production-safe quality determine the result.