The optimal summary interval for NPC conversations

The quadratic you didn't budget for#

The simplest way to give an NPC memory of a conversation is to resend the whole conversation every turn. It works, it's what most prototypes do, and it makes cost grow with the square of conversation length. A character who gets twice as chatty costs four times as much. This post derives the cost of full history, the cost of a common alternative (summarise the last $k$ exchanges, then drop them), and the interval $k^{\ast }$ that minimises spend. Then it checks the formula against an exact turn-by-turn count.

Code: research/context_budget.py. Prices are illustrative ($p_{\text{in}}=\$3$, $p_{\text{out}}=\$15$ per million tokens). Only their ratio $\rho =p_{\text{out}}/p_{\text{in}}$ affects the optimum.

Cost of full history#

Each turn sends a static prompt of $S=1800$ tokens (character sheet, rules, world state) plus every previous exchange of $m=160$ tokens, and gets $o=80$ tokens back. Turn $t$ carries $t-1$ previous exchanges, so over $n$ turns:

At 60 turns that's 391,200 input tokens, 72% of them repeated history. The static prompt is a rounding error by the end of a long conversation.

Summarise-and-flush, and its optimum#

Instead, keep at most $k$ raw exchanges. When the buffer fills, a summarisation call compresses the old summary plus those $k$ exchanges into a new summary of $s=150$ tokens (with $q=120$ tokens of instructions), and the buffer empties. The raw history in context then cycles from 0 to $k-1$ exchanges, $(k-1)/2$ on average. Approximating for $n\gg k$:

The middle term grows with $k$ (more raw history per turn); the last shrinks with $k$ (fewer summarisation calls). Setting $dC/dk=0$:

Note what's missing: $n$ and $S$. The optimal interval doesn't depend on how long the conversation runs or how big the character sheet is. It's a ratio of what a summary costs to what a raw exchange costs to carry. With our numbers, $k^{\ast }=\sqrt{2(150+120+5\cdot 150)/160}\approx 3.57$.

Checking it against an exact count#

Exact per-session cost at 60 turns for k = 1…40, counting every token turn by turn, including the cold start before the first summary exists.

The exact discrete optimum is $k=4$, next to the continuous $k^{\ast }\approx 3.57$, and the curve is flat near the bottom: anything from 3 to 5 is within 1% of optimal, and 6 is within about 2%. That flatness is useful in practice. You can choose $k$ for narrative reasons (summarise at scene boundaries) without paying much.

Cost per conversation as length grows. Summarisation wins from about 10 turns.

Summarisation starts paying off at around 10 turns and saves 57% at 60. For the long companion-style conversations where players talk for hundreds of turns, it's the difference between a viable game and one that can't be priced.

How prompt caching changes the answer#

The analysis above assumes every input token costs $p_{\text{in}}$. With prompt caching, history that was already sent last turn can be read from cache at roughly a tenth of the price. That makes carrying raw history ~10× cheaper, so the denominator $m$ shrinks by ~10 and

With a warm cache you should summarise far less often, roughly every 11 exchanges instead of every 4. This is a first-order estimate: it treats every carried token as a cache hit, which only holds if players reply within the cache TTL. Our caching post measures how often that's true.

What the model doesn't price: memory quality#

Every summary loses detail, and a character who forgets the name of your dog is a worse character. Two practical points from shipped titles:

• Summaries should be structured, not prose. A summary that records facts as key–value pairs ("player_promised: return lantern by day 9") survives repeated re-summarisation far better than a paragraph, which drifts a little each time.
• Pair summaries with retrieval. Store the raw exchanges outside the prompt and retrieve the few that matter. Summaries carry the gist; retrieval brings back exact quotes when the player asks "what did I say?". See NPC memory architecture.

Measure the trade-off directly. Run your eval suite at $k^{\ast }$ and at $2k^{\ast }$, and if quality scores don't move, take the cheaper one.

Reproduce it#

The full script is research/context_budget.py (Python 3.10+, numpy, scipy; fixed seeds, so you'll get the same numbers). This is the core of it:

def summarize_flush(n, k):
    tin = tout = hist = 0; has_summary = False
    for t in range(1, n + 1):
        tin += S + (s if has_summary else 0) + m * hist   # prompt for this turn
        tout += o
        hist += 1
        if hist == k and t < n:                            # compress and flush
            tin += (s if has_summary else 0) + k * m + 120
            tout += s
            hist, has_summary = 0, True
    return tin, tout

Run it with python3 research/context_budget.py; results land in research/out/, and the site's tables and charts are rendered from that JSON at build time. Set S, m and s from your own logs. Average them over a real playtest rather than guessing, because m varies a lot between chatty and terse players.

Frequently asked questions#