Prompt caching and the KV cache: how reusing a prompt prefix cuts latency and cost

Key points

• Within one request, prefill computes keys and values for the whole prompt once, and every decoding step reuses them.
• Prefix caching keeps those keys and values after a request ends and reuses them when a later prompt starts with the same tokens. It shortens prefill, not decoding.
• Matching is exact and runs from the first token: one changed character early in the prompt turns everything after it into a miss.
• OpenAI and Google cache eligible prefixes automatically; Anthropic caches up to the cache_control breakpoints you set, or automatically with one top-level setting.
• Order prompts from most stable to least: tools, system instructions, reference material, conversation history, then the new message.
• Measure the hit rate from the usage fields in every response, per prompt template, and watch time to first token split by hit and miss.

The KV cache inside one request

In every attention layer, a transformer turns each token into three vectors: a query, a key and a value. Attention, as defined in "Attention Is All You Need" (Vaswani and colleagues, 2017), scores a token's query against the keys of the tokens it may look at and uses those scores to mix their values. In a decoder that generates text, a token may look only at itself and the tokens before it, and generation is auto-regressive: each new token is produced from everything before it.

Without a cache, producing token 1,001 would mean recomputing keys and values for the first 1,000 tokens in every layer, and doing it again for token 1,002. Because earlier tokens never look at later ones, their keys and values do not change once computed. So the model keeps them: that store is the KV cache. Each generation step computes the query, key and value for the one new token, appends its key and value to the cache, and attends over everything stored.

This splits a request into two phases:

• Prefill: the whole prompt is processed in parallel, filling the cache with keys and values for every prompt token. Its work grows with prompt length, and it sets the time to the first output token.
• Decode: output tokens are generated one at a time, each reading the whole cache. Its work grows with the number of output tokens.

Prompt caching only ever saves prefill. The vLLM documentation says so plainly about its prefix caching: it does not reduce the time spent generating new tokens. A request with a long shared prompt and a short answer gains the most; a short prompt with a long answer gains little.

How big the KV cache gets

For each token, the cache holds one key and one value per layer per key-value head, each of the head's dimension. A quick estimate:

def kv_cache_bytes(tokens, layers, kv_heads, head_dim, bytes_per_value=2):
    # 2 tensors (key and value) per layer; 2 bytes per value in fp16 or bf16
    return 2 * layers * kv_heads * head_dim * bytes_per_value * tokens

# An illustrative model: 32 layers, 8 key-value heads of 128 dimensions each
per_token = kv_cache_bytes(1, layers=32, kv_heads=8, head_dim=128)
print(per_token)                                    # 131072 bytes = 128 KiB
print(kv_cache_bytes(8192, 32, 8, 128) / 2 ** 30)  # 1.0 GiB for one sequence

With those illustrative numbers, one 8,192-token sequence holds a gibibyte of keys and values. A server running many sequences at once is limited by how many caches fit in GPU memory beside the model weights. Models that share each key-value head across several query heads (grouped-query attention) have fewer key-value heads, and so a smaller cache.

The PagedAttention paper by Kwon and colleagues (SOSP 2023), which introduced vLLM, starts from that constraint. It describes how earlier serving systems wasted KV cache memory through fragmentation and duplicated copies, which limited batch size. PagedAttention borrows virtual memory and paging from operating systems: each request's cache is split into fixed-size blocks that need not sit next to each other in memory, found through a per-request block table. The authors report near-zero waste in KV cache memory, sharing of the cache within and across requests, and 2 to 4 times the throughput of FasterTransformer and Orca at the same latency.

Prefix caching: keeping the KV cache between requests

Blocks make the next step natural. A token's key and value depend only on the tokens up to it, so if two requests start with the same tokens, the keys and values for that shared start are identical. A server can keep them after the first request finishes and give them to the second, which then prefills only the part that differs.

vLLM's automatic prefix caching does this with hashes. Each full block gets a hash built from the hash of the block before it, the tokens in the block, and extra keys such as a LoRA adapter ID, a hash of any image the block refers to, or a per-request cache salt that keeps tenants apart. Because each hash includes its parent, a block can match only if everything before it matched too. Only full blocks are cached. Blocks no running request is using go to a free queue and are evicted least recently used first. The documentation names long-document question answering and multi-turn chat as the workloads that gain.

SGLang (Zheng and colleagues) uses a radix tree that maps token sequences to their keys and values. Instead of discarding the cache when a request finishes, it keeps both prompts and generated output in the tree and evicts the least recently used leaves first. Its scheduler orders waiting requests by the length of their matched prefix rather than first come, first served, so shared prefixes are reused before they are evicted. The authors report up to 6.4 times higher throughput than other inference systems across their benchmarks; RadixAttention is one of several optimizations behind that figure.

Prompt caching in the OpenAI, Anthropic and Gemini APIs

Hosted model APIs run the same mechanism on their own servers and expose it as prompt caching. The details differ by provider and change with new models; this table reflects each provider's documentation in November 2026.

Anthropic's cache covers the prompt in the order tools, system, messages, up to and including the block marked with cache_control. The usage block splits input three ways: cache_creation_input_tokens were written to the cache, cache_read_input_tokens were read from it, and input_tokens are the tokens after the last breakpoint, so the total input is the sum of the three. A prompt below the model's minimum is processed without caching and no error is returned. A breakpoint looks back at most 20 blocks for an earlier entry, so if one agent turn can add more blocks than that, add a second breakpoint earlier. Caches are never shared between organizations, and on the Claude API they are also isolated by workspace.

OpenAI routes each request to a machine based partly on a hash of the start of the prompt, and its guide says cache reuse requires the entire rendered prefix to match. A prompt_cache_key helps group requests that share a prefix, and the guide notes that traffic above about 15 requests per minute for one shared prefix and key can lead to overflow routing to other machines. The cached prefix includes instructions, tool definitions, conversation history with any images, and the structured output schema, so changing the schema mid-conversation breaks reuse.

Google's Gemini API advises putting large, common content at the beginning of the prompt and sending requests with a similar prefix in a short amount of time. Explicit context caches are created ahead of time with a lifetime and referenced by later requests; Google's Vertex AI documentation says they also incur storage charges for as long as they are kept.

Designing prompts for cache hits

Every provider matches from the first token forward, so a prompt is only as reusable as its earliest change. Order it from the most stable content to the least:

# Static prefix: identical on every request
tools       search_orders, order_status, refund_order   (fixed order, fixed text)
system      role, rules, output format                   (no dates, no user names)
reference   product catalog, policy documents            (versioned, changes rarely)
examples    worked question-and-answer pairs
------------------------------ breakpoint 1 ------------------------------
# Conversation: grows by appending, reused turn by turn
history     user and assistant turns, tool calls and results, in order
------------------------------ breakpoint 2 ------------------------------
# Dynamic suffix: different on every request, never reused
context     current date, user name and plan, passages retrieved for this turn
message     the new user message

The breakpoints are where an Anthropic request would carry cache_control; with automatic caching on any provider, the same order is what makes the prefix match. What breaks a hit:

• A timestamp, request ID or user name early in the prompt. Move it to the end, or into the latest user message.
• Tool definitions that change: a tool added per user, tools listed in a different order, a description edited. With Anthropic, a change to tool definitions invalidates the tools, system and messages segments. OpenAI recommends keeping tool definitions stable and using tool_choice or allowed tools to restrict use instead of removing tools.
• Serialization that is not deterministic: JSON keys in a different order, a set turned into a list, whitespace that varies. Serialize with sorted keys and fixed formatting.
• Rewriting history: summarizing, trimming or editing earlier turns changes the prefix from that point on. Append new turns, and when you must compact, do it rarely so the new prefix gets reused.
• Settings that sit inside the prefix. With Anthropic, changing tool_choice, adding or removing an image, or changing thinking parameters invalidates the messages segment. With OpenAI, the structured output schema is part of the prefix.
• Expiry and spread: a prefix not reused within its lifetime is gone, and heavy traffic on one prefix can be routed across machines.
• A prompt below the minimum length, which is simply not cached.

With Anthropic, put each breakpoint on the last block that is identical across the requests you want to share. Its documentation warns that a breakpoint on a block that changes every request, such as one holding a timestamp or the incoming message, means the hash never matches and every request pays for a cache write. When many requests share a fresh prefix, send one first and wait for its response to start before sending the rest, because the entry does not exist until then.

Measuring the hit rate

Every response reports its cached tokens, so the hit rate is a sum over your logs: cached input tokens divided by all input tokens. Measure it per prompt template or route, because one template that always misses can hide behind others that hit.

def hit_rate(usages, provider):
    cached = total = 0
    for u in usages:  # one usage object per response, as a dict
        if provider == "anthropic":
            read = u.get("cache_read_input_tokens", 0)
            write = u.get("cache_creation_input_tokens", 0)
            cached += read
            total += read + write + u.get("input_tokens", 0)
        else:  # OpenAI Responses API: input_tokens already includes cached ones
            cached += u.get("input_tokens_details", {}).get("cached_tokens", 0)
            total += u.get("input_tokens", 0)
    return cached / total if total else 0.0

logs = [  # three turns that share a 6,000-token prefix
    {"cache_read_input_tokens": 0, "cache_creation_input_tokens": 6000, "input_tokens": 120},
    {"cache_read_input_tokens": 6000, "cache_creation_input_tokens": 0, "input_tokens": 140},
    {"cache_read_input_tokens": 6000, "cache_creation_input_tokens": 0, "input_tokens": 95},
]
print(round(hit_rate(logs, "anthropic"), 3))  # 0.654

• Track the hit rate per template and per route, not only overall.
• With Anthropic, compare writes with reads. Many writes and few reads mean prefixes change or expire before they are reused.
• Split time to first token by hit and miss. The latency gain shows up there, not in total generation time.
• Compare the gap between requests on one prefix with its lifetime. If traffic is sparser than the lifetime, a longer lifetime or a warm-up request may help.

A hit rate near zero usually has one of three causes: the prompt is below the minimum length, something early in the prompt changes on every request, or requests for one prefix arrive too far apart for its lifetime.

Where prefix reuse helps most

• Agent loops. An agent calls the model many times per task with the same tools and instructions and a history that only grows, so each call can reuse everything before the newest tool result. See how LLM tool calling works.
• Chat assistants with long system prompts and multi-turn history; see AI chat assistant system design.
• Many questions over one long document, contract or codebase.
• Few-shot prompts with many examples, and extraction or classification jobs that run one fixed instruction over many inputs.

It helps least for short prompts, for prompts whose start differs each time, and for RAG prompts that put retrieved passages before the instructions: put the fixed instructions first and the passages after them.

Prompt caching is not response caching. The model still generates a fresh answer; only the prompt's processing is reused. Storing whole answers keyed by the request, or by an embedding of it, is a different technique with different risks; see caching strategies and text embeddings explained. For the other levers on spend, such as smaller models where they suffice, shorter outputs and batch processing, see how to reduce LLM costs.

If you want an AI agent or assistant built into your product, see our AI agent development service: the code lives in your GitHub from day one, and AI-assisted work is reviewed by an engineer.