The Inference Engine — From Tokens to Agents

In post #6 we ran a model the naïve way: one request, one token at a time. That's fine for understanding. It's a disaster for serving — running a model for millions of people, fast, without lighting money on fire.

Closing that gap is the job of the inference engine: the specialized software that sits between your request and the raw model and squeezes every drop of performance out of expensive hardware. This is where a huge amount of real engineering happens, and where projects like vLLM, TensorRT-LLM, SGLang, and others compete. Let's see what they're actually doing.

The two numbers that rule everything

Inference is a balancing act between two metrics that pull against each other:

Latency — how fast one user gets their answer. Usually measured as time to first token (how long until the response starts) and time per output token (how fast it then types).
Throughput — how many tokens total the system produces per second across all users. This is what determines cost per request.

You can often improve one by sacrificing the other. The art of serving is getting both as high as possible at once. And the reason it's hard comes down to the nature of the hardware.

The core problem: GPUs are starving

LLM inference runs on GPUs (and similar accelerators). A GPU has thousands of arithmetic units that can do math astonishingly fast — but they constantly need to be fed with data (the model's weights) from memory. For LLM decode, the bottleneck is almost never the math; it's memory bandwidth: shuttling those billions of weights from memory to the compute units.

Here's the killer detail. To generate one token, you must stream the entire model's weights through the chip. To generate that token for one user versus a hundred users costs almost the same in memory traffic — because you're loading the weights once either way. So:

Generating a token for 1 user wastes ~99% of the GPU. Generating a token for 100 users at once is nearly free per user.

That single fact drives the most important optimization in serving.

Batching: serve many users at once

Batching means processing many requests together so each expensive load of the weights does useful work for everyone in the batch.

One weight load — one token, or N tokens for the whole batch

The problem: in the real world, requests arrive at different times and have different lengths. If you wait to assemble a fixed batch, early arrivals sit idle; if a batch has one very long request, everyone waits for it.

The fix is continuous batching (sometimes "in-flight batching"): the engine manages a constantly shifting batch, slotting new requests in the moment a spot opens and evicting finished ones immediately — no waiting for the whole batch to complete. This one technique can multiply throughput several-fold and is a defining feature of modern engines.

The KV cache: don't redo work

Now the second big idea, and it requires remembering how attention works (post #4). At each decode step, every token attends to all previous tokens. Computing attention requires each previous token's key and value vectors.

Naïvely, at step 100 you'd recompute the keys and values for tokens 1–99 from scratch. Again at step 101. And 102. That's enormous, wasteful, repeated work.

The KV cache is the fix: compute each token's key and value once, store them, and reuse them for every future step. It turns generation from "recompute everything every step" into "compute only the new token and look up the rest."

Compute each key/value once — then only ever add the newest token

The KV cache is what makes generation tractable. But it comes with a catch that turns out to dominate serving economics:

The KV cache lives in precious GPU memory, and it grows with every token in every active conversation.

A long conversation or a big batch can have a KV cache that's larger than the model's own weights. GPU memory becomes the scarcest resource, and how you manage it determines how many users you can serve at once.

PagedAttention: memory management for the KV cache

Early engines allocated one big contiguous block of memory per request, sized for the worst case. Most requests didn't use all of it, so memory sat reserved and wasted — sometimes the majority of it. Fewer usable slots meant smaller batches meant lower throughput.

The influential idea here (introduced by vLLM, and named PagedAttention) borrows straight from how operating systems manage RAM. Instead of one big block per request, the KV cache is split into small fixed-size pages allocated on demand. A request uses only the pages it actually needs, and freed pages are instantly reusable by others.

Code

   Old way:  [■■■░░░░░░░] reserved-but-unused space wasted per request
   Paged:    [■][■][■]    allocate pages as needed; no waste; pack more requests

The payoff: far less wasted memory, so much larger batches, so much higher throughput. A nice bonus is that identical prefixes — say, a shared system prompt across thousands of requests — can share the same cached pages instead of each storing its own copy, saving even more memory and skipping repeated prefill work.

The other tricks engines use

A modern engine layers on more:

Prefix caching — remember the KV cache for common prompt prefixes (system prompts, few-shot examples, a long document being asked about repeatedly) and skip recomputing them.
Chunked prefill — break a huge prompt's prefill into chunks and interleave it with ongoing decode work, so one giant prompt doesn't stall everyone else.
Tensor / pipeline parallelism — when a model is too big for one GPU, split it across several, coordinating them so they act as one. Essential for the largest models.
Quantization-aware kernels — run the model in lower precision for speed (the whole subject of post #8).
Speculative decoding — use a small fast model to guess several tokens ahead and a big model to verify them in one shot (also post #8).

Why you should care even if you never run one

Even if you only ever call an LLM API, the inference engine shapes your experience and your bill:

Cost is downstream of throughput. The whole reason providers can charge what they do is that batching, KV caching, and paging let one GPU serve many users.
Latency behavior — slow first token, then fast streaming — is the prefill/decode split (post #6) running through the engine.
Why prompt caching saves you money — when a provider offers a discount for reusing a prompt prefix, that's prefix caching being passed on to you. Designing your prompts so the stable parts come first (post #9) can cut both latency and cost.
Why batch/async APIs are cheaper — they give the engine freedom to pack work efficiently, so the savings flow back to you.

The takeaway

The inference engine exists because naïve generation wastes almost all of an expensive GPU. By batching many users together (continuous batching), never recomputing past tokens (the KV cache), and managing cache memory like an OS manages RAM (paged attention and prefix sharing), engines turn an economically hopeless workload into a viable global service. Almost every number you see in an LLM pricing page is downstream of these ideas.

We've made the model run efficiently. The next frontier is making the model itself smaller and cheaper without losing its smarts.