How much inference capacity does an AI game need?

The question#

How much inference capacity does an AI-native game need to keep characters responsive? "Enough for peak" is the usual answer, and it hides the interesting part: how much headroom above average load you need depends heavily on how big the pool is. Queueing theory answers this exactly for a simple model, and a simulation tells us whether the simple model is safe to use.

Code: research/latency_queue.py. The workload numbers are assumptions; the method is the point.

The model#

Each active player sends a model request on average every $T=20$ seconds. Each request occupies one decode slot (a concurrent sequence in a continuously batched server) for a mean of $S=2.25$ seconds: prefill plus about 80 output tokens at ~40 tokens per second. With $N$ concurrent players, requests arrive at rate $\lambda =N/T$ and the offered load in Erlangs is

so 1,000 players offer $a=112.5$: on average that many slots are busy. Treating the pool as an M/M/c queue with $c$ slots, the probability that a request has to wait at all is the Erlang C formula, and the waiting time beyond that is exponential:

with $\mu =1/S$. Our target: fewer than 1% of requests wait more than 200 ms in the queue before decoding starts. We search for the smallest $c$ that meets it. (Computing $a^c/c!$ directly overflows beyond a few hundred slots. Our first version silently returned garbage at 10k players. The script uses the stable Erlang B recursion $B_k=a{B}_{k-1}/(k+a{B}_{k-1})$, then $C=c{B}_c/(c-a(1-B_c))$.)

Results#

Utilisation of decode slots when sized for P(wait > 200 ms) < 1%.

A pool serving 100 players needs 20 slots for 11.25 Erlangs of load: 56% utilisation, so nearly half the capacity you pay for sits idle as insurance. At 10,000 players the same latency target runs at 97%. This is the square-root staffing law:

Headroom grows with $\sqrt{a}$, not $a$, so relative headroom $\beta /\sqrt{a}$ shrinks as pools grow. The fitted $\beta$ falls from 2.6 to 0.43 because the target is fixed in milliseconds while service times are seconds: in a large pool, a slot frees up every few milliseconds.

Is M/M/c safe to use?#

Real service times aren't exponential. Output lengths cluster, so service times vary less. We ran a discrete-event simulation with lognormal service times (log-sd 0.5, same mean), 300,000 requests per run, at each slot count Erlang C recommended:

The simulation always beats the formula's prediction. Less variable service makes queues shorter, so Erlang C is conservative here: plan with it and you'll meet the target with a small margin. It would not be conservative if your output lengths were heavy-tailed (a character who occasionally monologues for 1,000 tokens). Cap max tokens per turn and the model stays honest.

What it means for a game#

• Pool across titles and regions where latency allows. Ten games with 1,000 players each, sized separately, need 1,320 slots. One shared pool for 10,000 needs 1,160. That's 12% less capacity for the same latency. This is the economic argument for a shared runtime, and it's why Ammighty Runtime exists.
• Small launches are the expensive ones per player. Early access with 300 concurrent players runs at 72% utilisation. Budget per-player inference cost at small scale accordingly, or use a shared pool from day one.
• Shorter service time buys more than more slots. Halving $S$ (a smaller model, fewer output tokens) halves $a$, and with it the slots needed. Combine with caching, which cuts prefill time as well as cost.

Limitations#

We model a slot as an independent server. On a real GPU, per-sequence throughput falls as the batch grows, so service time depends on load. Requests also aren't Poisson: scripted moments (a cutscene ending, a boss taunting) synchronise players, which creates bursts the model doesn't see. Use the formula for baseline sizing, then load-test with replayed telemetry. Finally, queueing delay is only one part of perceived latency; time-to-first-token and network round trip add to it. The LLM NPC handbook covers the full budget.

Reproduce it#

The full script is research/latency_queue.py (Python 3.10+, numpy, scipy; fixed seeds, so you'll get the same numbers). This is the core of it:

def erlang_c(c, a):
    b = 1.0
    for k in range(1, c + 1):          # stable Erlang B recursion
        b = a * b / (k + a * b)
    return c * b / (c - a * (1 - b))

def p_wait_gt(c, lam, mu, t):
    return erlang_c(c, lam / mu) * math.exp(-(c * mu - lam) * t)

c = floor(a) + 1
while p_wait_gt(c, lam, mu, 0.2) > 0.01:
    c += 1

Run it with python3 research/latency_queue.py; results land in research/out/, and the site's tables and charts are rendered from that JSON at build time. Swap in your own think time and service time, measured end to end on your serving stack under load, not from a single request. The discrete-event simulator des() in the same file lets you try other service-time distributions.

Frequently asked questions#