Why dialogue is a caching problem#
An NPC's prompt is mostly the same every turn: personality, rules, world state, the conversation so far. Providers that support prompt caching charge much less to re-read a prefix they've already processed. The catch is a time-to-live: if the player takes longer than the TTL to reply, the cache expires and you pay a premium to write it again. Players are not API clients. They read, think, wander off to the inventory, go make tea. This post measures how much caching saves under realistic think-time patterns, and when the longer, pricier TTL is worth it.
We use Anthropic's published multipliers relative to the base input price: cache reads at 0.1×, writes at 1.25× for a 5-minute TTL and 2.0× for a 1-hour TTL, with the TTL refreshed on each hit. Code: research/caching.py.
The break-even hit rate#
Let be the fraction of turns that hit the cache, the write multiplier and the read multiplier. The expected cost of the cached prefix, relative to sending it uncached, is
Caching saves money when , i.e. when
which is for the 5-minute cache and for the 1-hour cache. A 5-minute cache pays off if just over one turn in five hits. The 1-hour cache needs more than half, but long gaps rarely break it.
Comparing the two TTLs over a session of turns, where the first turn is always a write: if the 1-hour cache hits almost every time, it beats the 5-minute cache when the 5-minute miss rate exceeds
For a 60-turn session that's a miss rate above about 1.1%. Almost any real player crosses it.
Simulation#
We simulated 20,000 sessions of 60 turns for each of six think-time profiles. The prompt has a 2,400-token cacheable prefix and a 300-token uncached suffix (the new player line and dynamic state). Gaps between turns are lognormal around the profile's median (log-sd 1.0), and 8% of gaps are breaks (menus, AFK) with a median of 8 minutes. Those breaks are an assumption, but anyone who has looked at session telemetry knows they exist.
Total input cost per turn (cached prefix + uncached suffix) relative to sending everything uncached.
| Median think time | 5-min hit rate | 5-min cost | 1-hour hit rate | 1-hour cost |
|---|---|---|---|---|
| 5s | 92.7% | 28% | 98.3% | 23% |
| 15s | 92.6% | 28% | 98.3% | 23% |
| 30s | 91.7% | 29% | 98.3% | 23% |
| 60s | 87.8% | 32% | 98.3% | 23% |
| 120s | 76.4% | 44% | 98.3% | 23% |
| 240s | 55.5% | 66% | 98.0% | 23% |
What we found#
Caching cuts dialogue input cost by roughly three quarters. At a 30-second median think time, the 5-minute cache brings input cost to 29% of uncached and the 1-hour cache to 23%. The floor is set by the uncached suffix: 300 of 2700 tokens are new every turn and can never be cached.
Slow, thoughtful players break the short cache. At a 4-minute median, typical of mystery and strategy games where players read clues or plan between lines, the 5-minute hit rate falls to 55% and cost rises to 66%. The 1-hour cache barely notices (23%).
Breaks matter more than average pace. Even our fastest profile (5-second median) has a 5-minute hit rate of only 93%, because 8% of gaps are long breaks. That's well above the 1.1% threshold, so the 1-hour cache wins at every pace we tested.
Making the prefix cacheable#
None of this works unless the prefix is byte-identical between turns. The common ways games break their own cache:
- Timestamps or turn counters at the top of the prompt. Move every volatile field to the end.
- Re-serialising world state in a different key order. Use a canonical serialiser.
- Retrieving memories into the middle of the prompt. Retrieved snippets change every turn. Put them after the stable block, not inside it.
- One prompt per NPC variant. Share a common prefix (world rules, tone bible) across characters, then branch. Providers cache by prefix, so a shared head gets reused across every character in the scene.
Prefix ordering interacts with summarisation, too: a warm cache makes carrying raw history cheap, which moves the optimal summary interval from about 4 exchanges to about 11.
Limitations#
Prices, TTL options and minimum cacheable lengths differ between providers and change over time. The 1-hour option isn't available on every platform, and self-hosted inference servers have their own eviction policies driven by memory pressure, not a fixed TTL. The think-time distributions are assumptions, so replace them with your telemetry; the script takes any gap distribution. We also assume one cache entry per conversation. Multiplayer scenes where several players talk to one NPC share entries and do better.
Reproduce it#
The full script is research/caching.py (Python 3.10+, numpy, scipy; fixed seeds, so you'll get the same numbers). This is the core of it:
G = gaps(median_s, SESS * (TURNS - 1)).reshape(SESS, TURNS - 1) # seconds between turns
hits = G < ttl # TTL refreshes on every hit; turn 1 always writes
hit_rate = hits.sum() / (SESS * TURNS)
prefix_mult = (hits.sum() * r + (SESS * TURNS - hits.sum()) * w) / (SESS * TURNS)
total_mult = (prefix_mult * P + U) / (P + U)Run it with python3 research/caching.py; results land in research/out/, and the site's tables and charts are rendered from that JSON at build time. The most valuable change is replacing gaps() with think times from your own telemetry. Export the timestamps of player inputs and feed the differences straight in.
Frequently asked questions#
Does prompt caching work for game dialogue?
Yes. In our simulation it cut input cost to 23–29% of uncached for typical think times, as long as the stable part of the prompt comes first and is byte-identical between turns.
Should I use a 5-minute or 1-hour cache TTL?
For sessions longer than a few dozen turns with real-world breaks, the 1-hour TTL usually wins: its higher write cost is amortised and it survives pauses. The 5-minute TTL wins only for short, fast, uninterrupted exchanges.
What is the break-even cache hit rate?
h* = (w−1)/(w−r): about 22% for a 1.25× write / 0.1× read cache and about 53% for a 2× write / 0.1× read cache.
Sources#
- Helicone docs, "Prompt caching" (summary of Anthropic cache pricing: read 0.1×, 5-min write 1.25×, 1-h write 2×). docs.helicone.ai/gateway/concepts/prompt-caching