The LLM NPC handbook: architecture to launch

Architecture: the model proposes, the game decides#

The most important design decision is where game state lives. If inventory, quest flags, relationships and rules live in the prompt and the model is trusted to track them, they will drift. A character will hand over a key the player already has, or forget a betrayal. The pattern that works:

1. Parse intent. Turn the player's free-form input into a structured intent (ask_about, threaten, offer_item) with a small model or classifier.
2. Read state. Pull relevant game state and memories from your own systems.
3. Generate. Ask the character model for a line and a structured proposal ("give_item: lantern", "trust: −0.1").
4. Validate. Game code checks the proposal against rules and either applies it or rejects it.
5. Guard. A tone and rating check runs on the line before display; failures fall back to authored lines.

The model never mutates state directly. That one boundary makes characters debuggable, makes saves deterministic, and makes cheating via prompt injection mostly harmless. The worst a jailbreak can do is make a character say something odd, and the guard handles that.

Memory: what to carry and what to fetch#

Characters need two kinds of memory: the current conversation and everything before it. Carrying the whole conversation every turn makes cost grow with the square of its length:

where $S$ is the static prompt and $m$ the tokens per exchange. Summarising every $k$ exchanges makes it linear, and the cost-optimal interval $k^{\ast }=\sqrt{2(s+q+\rho s)/m}$ doesn't depend on conversation length. We derive and verify it in the optimal summary interval.

Long-term memory is a retrieval problem: store observations outside the prompt and fetch the few that matter for this turn. The scoring scheme from the Generative Agents paper (recency × importance × relevance) is a good default. NPC memory architecture covers it, along with why you should put retrieved memories at the end of the prompt rather than in the middle.

Latency budget#

Perceived latency is the time from the player finishing their line to the character starting to respond. It's a sum, and every term is yours to manage:

A workable budget for conversational play, at the 95th percentile:

Above roughly half a second, conversation starts to feel like waiting. Three tactics buy perceived speed without new hardware: stream output and start speaking or animating on the first clause; bark an authored filler line ("Hm. The ferry…") while generation runs; and pre-generate likely replies while the player is still reading. Queueing is the term people forget. Sizing it properly is the subject of inference capacity planning.

Model routing#

Most turns don't need your best model. Route each turn to a small or large model based on a cheap classifier. If a fraction $\pi$ of turns can go to the small model, and the classifier itself costs $c_r$ per turn:

Routing beats always-large when $\pi (c_{\text{large}}-c_{\text{small}})>c_r$. With a small model at a tenth of the large model's cost and a classifier at 2%, you only need to route $\pi >0.022$, a bit over 2% of turns. In practice 50–80% of turns (greetings, small talk, acknowledgements, repeated questions) are routable. The risk isn't cost, it's quality: misrouting a plot-critical line to the small model. Bias the classifier toward escalation and evaluate both sides of the boundary.

Cost per player-hour#

The planning number for any AI-native game is cost per player-hour:

Count every call, including intent parsing, summarisation, guards and off-screen simulation, not just dialogue. Typical optimised values run from about $0.01/hour for conversation-light games to over $1/hour for heavy multi-agent simulations; our inference cost guide has ranges and a calculator. The levers, in rough order of impact: prompt caching (cuts dialogue input cost by about three quarters in our simulation), summarisation, routing, fine-tuned smaller models and on-device inference.

Whether a given $\kappa$ is affordable depends on the business model, and premium and free-to-play fail differently. In free-to-play, inference takes a fixed share $\varphi =\tau \kappa /(60\cdot \text{ARPDAU})$ of revenue regardless of retention; in premium, your heaviest players can cost more than they paid. Retention curves and the inference bill works through both.

Tone guards and safety#

Players will try to break your characters. Some want a funny clip, some want to see what's behind the curtain, a few want something harmful. Defend in layers:

• Input screening for clear abuse and prompt-injection patterns. Don't over-block; players type strange things legitimately.
• A strong character frame: a tone bible, banned topics, and example refusals in character ("The harbourmaster doesn't talk politics, and he tells you so").
• Output classification against your rating envelope before display, with authored fallback lines for every failure category.
• Server-side guardrail updates so you can respond in hours without a client patch.

Design the fallback to be in-world. A character who changes the subject is fine. One that shows an error is a bug report and probably a clip. For store and rating requirements, see AI disclosure and ratings.

Evaluation: testing the non-deterministic#

You can't unit-test a line of dialogue, but you can test the distribution. Three layers:

1. Deterministic tests on everything around the model: intent parsing, validation, state updates. Most bugs live here.
2. Rubric evals: a fixed suite of a few hundred scenarios, generated responses scored (by a judge model plus periodic human calibration) on staying in character, lore consistency, rule-following and rating compliance. Run it on every prompt or model change; a drop fails the build.
3. Synthetic playthroughs: agent players with different styles run whole sessions overnight; humans review the outliers.

Track a small number of metrics over time: in-character rate, guard intervention rate, fallback rate, and median and p95 latency. When any of them moves, you want to know which commit moved it.

Offline and degraded modes#

Every networked AI game will see its model unavailable at some point: an outage, a player on a train, a region with bad connectivity. Plan a ladder of fallbacks, from best to worst: an on-device small model for routine turns; cached responses for common inputs; authored lines keyed to intent; finally, an in-world reason the character won't talk right now. Test the ladder like any other feature. Pull the network cable in QA and play for an hour.

Frequently asked questions#

Sources#

1. Park et al., "Generative Agents: Interactive Simulacra of Human Behavior", arXiv:2304.03442 (2023). arxiv.org/abs/2304.03442