Prompt caching cost savings: what the pricing pages don't tell you

Every major LLM provider now offers prompt caching. Anthropic, OpenAI and Google all discount cached input tokens by 50-90% compared to standard rates. The numbers on the pricing pages look transformative. But if you have actually tried to model what caching will save your team, you have probably found the gap between theory and practice is wider than the marketing suggests.

I have spent the last few weeks working through how caching behaves across different production workloads. This post covers what caching actually costs, what hit rates different workloads realistically achieve, and where the savings disappear.

How caching pricing actually works

Each provider implements caching slightly differently, but the economic shape is the same. You pay a premium to store content in the cache (a cache write), then a steep discount to reuse it (a cache read). The write premium and the read discount define your break-even point.

Anthropic

Anthropic offers two cache durations: 5-minute and 1-hour TTLs.

• Cache write (5-minute): 1.25x your base input price
• Cache write (1-hour): 2x your base input price
• Cache read (hit): 0.1x your base input price

For Claude Opus 4.8 at $5/MTok input, a cache read costs $0.50/MTok. A 5-minute cache write costs $6.25/MTok. You break even on the 5-minute TTL after one cache read. For the 1-hour TTL ($10/MTok write), you need two reads to break even.

The interesting detail: Anthropic automatically advances the cache breakpoint as conversations grow. In a multi-turn chat, the system caches the conversation history up to the latest turn, which means later turns get progressively cheaper.

OpenAI

OpenAI breaks pricing into short context and long context. For the latest models, cached input runs at roughly 10% of the input price across the board. For gpt-5.6-sol ($5/MTok input), cached input is $0.50/MTok. For gpt-5.6-luna ($1/MTok), it is $0.10/MTok.

OpenAI also has a separate cache writes column at 1.25x the input rate, similar to Anthropic's 5-minute TTL writes. Caching is automatic for eligible prompts -- you do not need to set explicit breakpoints.

Google Gemini

Google charges $0.00 per hour to store a context cache (no storage fee). You only pay for the initial cache write at the standard input rate, and subsequent cached reads at roughly 50% of the standard rate -- less aggressive than Anthropic or OpenAI's 90% discount, but there is no write premium and no storage cost. For long-running caches (hours or days), this can work out cheaper overall.

Real hit rates by workload type

The pricing only tells you the per-token math. The real question is what hit rate your workload will actually see. I have gathered data from production teams, community benchmarks, and our own testing at Cost.

Chat sessions: 60-80% hit rate

Multi-turn chat is where caching shines. A user sends a message, the assistant responds, the conversation grows. With Anthropic's automatic caching, the system prompt and conversation history up to the last turn get cached. Each new turn reads the cache for the history and only pays uncached rates for the new user message.

In practice, chat workloads see a 60-80% cache hit rate on input tokens once the conversation passes 3-4 turns. Short sessions (1-2 turns) see much lower rates, but for the kind of deep chat sessions that generate large bills, caching is extremely effective.

Real example: a customer support chatbot with a 4,000-token system prompt and average 8-turn conversations. Roughly 75% of input tokens become cache reads after the warmup turn. Cost per conversation drops by about 60% compared to uncached pricing.

Code generation: 30-50% hit rate

Code generation prompts tend to be more varied. The system prompt and repository context cache well, but the specific code request changes each time.

If you include a large repository context or codebase summary in every request (10,000-20,000 tokens of shared context), that portion caches effectively. But the user prompt itself -- the specific function they want written or the bug they want fixed -- varies widely and rarely hits cache.

The sweet spot is agentic coding loops, where the same assistant works through multiple files in a session. Each turn adds context but the shared understanding of the codebase stays in cache.

RAG pipelines: 10-30% hit rate

Retrieval-augmented generation workloads have the worst caching profile. Each query fetches different documents from the vector store. The system prompt caches, but the document context changes every time.

If you include a large static knowledge base or instruction set in every query (common in enterprise RAG), that portion caches fine. But the retrieved chunks are unpredictable. Some teams pre-pend a fixed system prompt of 2,000-3,000 tokens and see around 20% cache hits on total input. The rest is uncached retrieved content.

Agent loops: 40-60% hit rate

Autonomous agents that maintain a session across multiple tool calls and reasoning steps fall between chat and code generation. The agent's instructions, tool definitions and accumulated state cache well. But each step introduces new observations from tool outputs.

Hitting 50-60% is realistic for long-running agent sessions (10+ steps). Short agent loops (2-4 steps) barely benefit because the cache has not accumulated enough reusable content before the session ends.

The hidden gotchas

Cache warmup costs

Teams implementing caching often focus on the read discount and forget the warmup phase. The first request in a session pays the write premium. If your sessions are short (1-2 requests), you might pay more for the write than you save on the read.

A concrete example: with Anthropic's 5-minute TTL on Opus 4.8, a single 10,000-token write costs $0.0625. One read costs $0.005. You saved $0.0575 on the second call. But a second call using the same tokens without caching would have cost $0.05. The actual saving per call beyond the break-even point is about $0.045. Not nothing, but much less than the 90% headline discount suggests.

Eviction and TTL

Caches expire. Anthropic's 5-minute cache is evicted if not refreshed within 5 minutes. If your users send messages 7 minutes apart, you pay the write premium on every message. The 1-hour TTL costs more per write but is more forgiving for irregular traffic.

OpenAI and Google have similar constraints. The practical lesson: caching benefits bursty workloads less than steady conversation streams. If your traffic pattern has long gaps between requests to the same session, the cache expires before it pays off.

Variable-length contexts

Caches are prefix-based. If you change the beginning of your prompt (different system prompt, different instructions), the cache invalidates. This is a subtle but important constraint. If your application serves multiple use cases from the same model endpoint with different system prompts, you fragment the cache and reduce hit rates.

When caching barely helps (or costs more)

Three workload profiles where prompt caching is not worth the implementation effort:

One-shot prompts. A single request with no follow-up. You pay the write premium and never read the cache. You would have been better off with standard pricing.

Highly varied prompts. If every request uses a completely different system prompt and user context, the cache never builds. The cache write cost is a tax with no benefit.

Very short sessions. Conversations of 1-2 turns. Even if you get one cache read, the saving on that read barely offsets the write premium on the first turn.

Caching and model swapping: how the two compound

Prompt caching is one lever for reducing inference cost. Model swapping -- replacing an expensive frontier model with a cheaper one for certain requests -- is another. The two compound well, and this is where Cost's approach comes in.

A common pattern: use a cheaper model (Claude Haiku 4.5 or gpt-5.4-nano) for straightforward requests where caching handles most of the context, and route complex or novel requests to a more capable model. Caching reduces the cost of the cheap-model calls further, and the expensive model only handles the edge cases where you need the extra reasoning.

Verification is the missing piece. Swapping a model or relying on cached responses changes the output quality. You need a way to check that cheaper cached paths still produce acceptable results. That verification step is what Cost handles -- testing whether the output quality holds when you switch models or change caching configurations, before you ship the change to production.

What hit rates are you seeing with prompt caching in production?

The numbers I have shared are based on public benchmarks, vendor documentation, and our own analysis at Cost. But production caching behaviour depends heavily on your specific workload shape. If you are running prompt caching in production, I would be interested in what hit rates you are actually seeing and whether the pricing page numbers matched your experience.

If you are still evaluating whether caching makes sense for your use case, try modelling it with your actual prompt sizes and session lengths before enabling it. The 90% discount headline sounds transformative, but the real savings depend on your workload pattern, not the pricing page.