Prompt caching, explained

Open the request log of any coding agent and one thing jumps out: every turn resends the entire conversation. The system prompt, the tool definitions, every file the model has read, every command it has run. All of it, repeated, on every request. A one-hour session can easily push the same few hundred thousand tokens through the API fifty times.

Imagine if you had to pay base API prices on those repeated tokens, every single time. Prompt caching is the reason you don't. Providers that support it serve the repeated part of your prompt at a fraction of the normal input price, often around a tenth. If you're running autonomous or deep-reasoning agents, this one mechanic decides whether their sessions costs cents or tens of dollars, so it's worth knowing exactly how it works, how to make the most of it and what inevitably breaks it.

A transformer doesn't have memory between API calls. Each request is a blank slate: the model ingests the full prompt, token by token, and builds up an internal working state. For every token, it keeps a set of vectors (the keys and values in attention) that every later token looks back at. This state is called the KV cache, and computing it is most of what you pay for on the input side.

Here's the thing, that computation is deterministic for a fixed prefix. If request n + 1 starts with exactly the same tokens as request n's (same system prompt, same tool definitions, same conversation so far), the internal state after reading them is byte-for-byte identical. Recomputing it is pure waste.

So providers stopped recomputing it all together. After serving your request, they keep the KV cache around for a few minutes. If another request arrives that starts with the same token prefix, they load the saved state and only compute the new suffix. You get billed accordingly: a discounted rate for the reused prefix, the full rate for whatever's new.

It's very important to understand that prefix means the match runs from the very first token and stops at the very first difference. Caching never skips over a changed token to reuse what comes after it. Change one word in your system prompt and everything downstream of it is a cache miss. And a few minutes means this is a hot cache, not storage. Let a session go idle through the TTL and the next turn pays full price once to rebuild it.

Providers expose this in two ways.

Implicit caching is fully automatic. The provider fingerprints incoming prompt prefixes and reuses whatever state it still has, with no API changes and nothing to opt into. OpenAI works this way (kicking in past a minimum prompt length), as do DeepSeek and several others. The discount just appears on your bill as a cached_tokens line.

Explicit caching puts you in control. Anthropic is the canonical example: you place cache_control breakpoints in the request to say "cache everything up to here," and you pay a small surcharge to write the cache in exchange for the discount every time you read it.

{
  "system": [
    {
      "type": "text",
      "text": "You are a coding agent. <...the long stable part...>",
      "cache_control": { "type": "ephemeral" }
    }
  ]
}

Implicit caching is zero effort and impossible to get wrong, but you can't force it, you're trusting the provider's matcher.

Explicit caching costs a little thought (and a write premium), but you know exactly what's cached and when, which matters when repeated cache misses on 200k tokens is real money.

Exact numbers can't be provided as they're always changing, so always check your provider's pricing page, but the ratios are consistent across the industry:

The effect compounds in agent workloads. Say the transcript has grown to 100k tokens and you run twenty more turns. Without caching, that's roughly 2M input tokens at full price. With a ~10% cached rate, the same twenty turns bill closer to around 300k tokens' worth. The cached transcript costs a tenth, and only each turn's genuinely new tokens (a few hundred each) pay full price.

This is also the fine print when comparing providers. An open-weight host with rock-bottom list prices but no prompt caching can end up more expensive for agents than a pricier provider with a good cache. Sticker price tells you what one turn costs while caching decides what a session costs.

Because matching is prefix-only, prompt layout is cache strategy. A few rules cover most of it:

1. Stable content first. System prompt, then tool definitions, then conversation history, with anything per-request at the very end. Every major agent framework uses exactly that order.
2. Append, don't edit. A conversation that only grows at the tail re-caches beautifully, since each turn just extends the prefix. Rewriting earlier history (summarizing it, reordering tools, renumbering messages) invalidates everything after the edit.
3. Keep volatility out of the prefix!! A timestamp, a request ID, or a "current date" line near the top of the system prompt is a self-inflicted 100% miss rate. If it must exist, put it at the end.
4. Mind the clock. Caches live for minutes. Agents in an active loop stay hot on their own; a human who reads for ten minutes between messages may eat a full-price rebuild. Some providers sell longer TTLs.
5. Watch cached_tokens in responses. Every provider that caches reports it in usage metadata. If the number isn't climbing turn over turn in an agent session, something upstream is churning your prefix.

It's worth knowing a few limits of prompt caching before you architect around them. The cache is scoped only to you: prompts are not shared across accounts, so there's no privacy leak, but also no benefit from other people's identical prompts. It only helps the input side as output tokens are always full price and always regenerated, obviously. And it only caches reading, not reasoning as the model isn't remembering conclusions between calls, just the KV Cache of the ingested tokens it has seen before.

One more practical note for the open-weight world is that caching is a property of the serving stack, the inference provider, not the model itself. The same Qwen or GLM model checkpoint costs wildly different amounts to run an agent on depending on whether the host has enabled prefix caching or not. If you're picking a provider for agentic workloads (as in our walkthrough of running Claude Code on open-weight models), "do you cache prompts?" should be one of your top questions, right next to the price per million tokens.

The Inference Hub is a global community for AI practitioners. If token bills and cache hit rates are your idea of an interesting conversation, you're in the right place.