Get in touch

Prompt Caching: How to Cut LLM Costs by up to 90%

Same bot, two different worlds

Picture a customer support bot with 100,000 tokens of documentation. Two queries, the same system prompt. Without caching: input costs of ~$150 a day at 500 requests on Claude Sonnet 4.6, and the first token shows up after 4–5 seconds. With prompt caching: ~$15 a day and a TTFT of 1–2 seconds. The difference isn't magic, just the reuse of previously computed state.

Anthropic offers this through explicit block markers (cache_control); OpenAI does it automatically. In both cases a cache read costs ~0.1× the standard input price (a 90% saving), and latency on long prompts drops by 67–80%.

Where does the cost come from? Prefill and the KV cache

Before a model produces the first token of its answer, it first has to "read" the entire input. This is the prefill phase. For every token in a 100,000-token prompt, the model computes vector representations (the key and value states, or KV) and runs self-attention: each token attends to all the ones before it to understand the context. The cost of this grows quadratically with prompt length: for 100K tokens, that's hundreds of billions of matrix multiplications. Only after prefill finishes does the model start generating output. That's why Time To First Token (TTFT) is often much longer than generating the tokens that follow. What the user is mostly waiting for isn't the answer being "written", but the prompt being "read".

Within a single request, the model uses a KV cache: once the key-value states for a token are computed, it doesn't recompute them at later steps. Prompt caching extends this mechanism beyond the boundary of a single request: if the next query shares an identical prefix, the KV states are read from memory instead of being computed from scratch. The Prompt Cache paper (MLSys 2024) shows TTFT reductions of 1.5× to 10× on GPU and 20× to 70× on CPU, with no change to the model's parameters.

A map of the solution space

To avoid getting lost in the terminology, it helps to look at two axes:

The level axis:

  • Intra-request: KV caching within a single query (standard in vLLM, TGI).
  • Inter-request: sharing computation across queries in a short window (the prompt caching APIs: Anthropic, OpenAI, Google Vertex AI).
  • Inter-session: durable caching across user sessions (e.g. Vertex AI explicit caching, vLLM APC on-premise).

The control axis:

  • Implicit: the provider detects repetition automatically (OpenAI, Google Vertex AI by default). Zero configuration, no control.
  • Explicit: the developer marks parts of the prompt for caching (Anthropic's cache_control). Full cost control and deterministic behavior.

Google Vertex AI offers both: implicit context caching (automatic, a 90% discount on Gemini 2.5+) and explicit context caching (caches created explicitly, with a guaranteed discount).

Prompt caching in practice: prompt layout matters

The biggest mistake in caching rollouts concerns prompt layout. A prefix cache matches an identical prompt beginning from the very first token. Put anything that changes at the start (a timestamp, a session ID, a short per-user intro) and the whole prefix changes, guaranteeing a cache miss.

Bad prompt layout: changing metadata at the start breaks the whole prefix and the cache hits 0%. Good layout: a stable system prompt first, the changing parts at the end, and the cache hits ~95%

The rule is simple: everything that doesn't change (system prompt, documentation, tool definitions) goes at the start. Everything that changes (timestamp, session ID, retrieved documents, user query) goes at the end.

An experiment from August 2025 shows the scale of the problem: a stable prefix gave a median TTFT of 953 ms, a changing one 2,727 ms. A 65% difference. The most common production mistake is placing dynamic metadata before the system prompt. Move everything volatile to the end, and the 90,000 tokens of system prompt form an identical prefix, with the cache hit rate jumping from 0% to 80–95%.

Three players, three cost philosophies

ProviderControlCache write / creationCache readMin. tokensStorage costTTL
AnthropicExplicit (cache_control)1.25× base (5 min) or 2× (1 h)0.1× base512–4096 (model-dependent)None5 min or 1 h
OpenAIImplicit (automatic)No extra charge0.1× base (GPT-5.x) to 0.5× (GPT-4o)1024None5–10 min (older models), 24h by default (GPT-5.5+)
Vertex AI (Gemini)Implicit + ExplicitStandard input price (creation)0.1× base (Gemini 2.5+, 90%) or 0.25× (Gemini 2.0, 75%)1024–4096 (model-dependent)Yes: per million tokens / hour (e.g. Gemini 3.1 Pro: $4.50/MTok/hr, Flash-Lite: $0.50/MTok/hr)Implicit: up to 24h. Explicit: configurable (default 1h)

The key way Vertex AI differs from the competition: paid storage on explicit caching. Anthropic and OpenAI charge nothing for holding the cache; you pay only for writes and reads. On Vertex AI, creating an explicit cache means paying three times: (1) the standard input price at creation, (2) storage for every hour the cache is held, (3) the discounted read on use. That changes the economics: with large contexts and sparse traffic, the storage cost can eat the savings. For frequent queries within a single hour it still pays off, but it demands tighter TTL management.

Semantic caching: a separate layer

Prefix caching optimizes computation: don't compute the same thing twice. Semantic caching optimizes relevance: don't ask the model the same question twice. Research suggests 31% of LLM queries are semantic repeats. But since users rarely repeat questions word for word, a straight token-by-token comparison almost never matches.

Semantic caching takes a different route: the new query is turned into a vector (an embedding) by an embedding model, then searched against previous queries in a vector DB by cosine similarity. If the similarity crosses a threshold, the system returns the stored answer without calling the LLM.

But that introduces a risk: two different questions can be semantically close and still need different answers. Well-tuned systems hit 0.8% false positives in benchmarks, which sounds acceptable. But a bank case study shows the same system hitting 3.8% on real production traffic, nearly 5× more. The lab's 0.8% gives you a false sense of safety.

In practice, the two mechanisms work together. The prefix cache lowers the computation cost at the model level, while the semantic cache eliminates unnecessary LLM calls for questions that have come up before. Even when the semantic cache misses, the prompt can still be cheaper thanks to prefix caching, because the system prompt is already computed.

Four questions before you deploy

Rather than rolling out caching blindly, walk through four decision questions.

Question 1: Do you have a long, stable prefix?

If your system prompt + documentation + tool definitions exceed 5,000–10,000 tokens and change less often than once an hour, prompt caching will save you money. If the prompt is short (< 1,000 tokens) or dynamic, the cache write cost can outweigh the savings.

Question 2: How high is your traffic, and will the cache write pay for itself?

Cache writes cost money, though not with every provider. With Anthropic's explicit caching you pay 1.25× base input (5-min TTL) or 2× base input (1-h TTL) for the first processing of a block. To break even, you need at least one subsequent cache hit (at 5 min) or two (at 1 h). With OpenAI's implicit caching there's no extra write cost, but retention depends on the model: older models (up to and including GPT-5.4) hold the cache for 5–10 minutes by default with an extended option up to 24h, while the newest models (GPT-5.5 and later) default to 24h retention.

If you're considering a semantic cache, remember the vector search cost (~30 ms). It only pays off at a hit rate of ≥ 15–20% per category. High-volume categories reach 40–60%, but rare specialist queries sit at 5–15%, so the average hit rate can fall below the threshold.

Question 3: Are the queries deterministic or creative?

Prefix caching shines on deterministic tasks: RAG, document analysis, generating code from a spec, responses driven by tool calls. On creative tasks (story writing, brainstorming) users rarely repeat identical or even similar queries, so the cache hit rate will be low.

Question 4: What are the consequences of a wrong answer?

This one is mainly about the semantic cache. If a wrong answer means losing a customer's trust, a medical error, or a financial loss, the semantic cache needs an extra verification layer on top. In cases like that, prompt (prefix) caching may turn out to be the safer, more predictable optimization.

Summary

Don't turn on prompt caching until you've answered the four questions: do you have a stable prefix, what does your traffic look like, are the queries deterministic, and what does an error cost.

If the prefix is long and stable, the traffic sufficient, and the queries repetitive, a real 90% cut in input costs and a 67–80% cut in TTFT is on the table. But those numbers won't arrive on their own. Anthropic demands deliberate prompt design and a paid cache write. OpenAI works automatically but gives you no control over what gets cached. Vertex AI adds storage to the bill, which with sparse traffic can eat the entire saving.

The biggest cost lies in the mistakes: dynamic content at the start of the prompt takes the hit rate from 95% to 0%, and the semantic cache's 0.8% lab false positive rate turns out closer to 3.8% in production. Without prompt design, traffic analysis and per-category monitoring, the costs won't come down.

Sources