Inference: How a Model Generates Text

Training is done. The weights are frozen. Now you type a question and the model answers. That act — running a trained model to produce output — is inference, and it's what happens billions of times a day around the world.

It's also full of choices that directly shape what you experience as the model's "creativity," "reliability," and "speed." Let's open it up.

One token at a time, on repeat

Recall the autoregressive loop from post #2. Here it is again, now that we know what's inside the model() call (the whole transformer from post #4):

   prompt: "Write a tagline for a coffee shop:"

   step 1:  run the model over the prompt        → predict "Brew"
   step 2:  run it over prompt + "Brew"           → predict "ed"
   step 3:  run it over prompt + "Brewed"         → predict " to"
   step 4:  run it over prompt + "Brewed to"      → predict " perfection"
   ...
   until:   the model predicts a special "stop" token, or hits a length limit

Two consequences fall out of this immediately, and they explain a lot:

• The model generates left to right and can't edit. Once a token is emitted, it's committed; the model can't go back and revise an earlier word. This is why models sometimes write themselves into a corner and then awkwardly recover — they're improvising forward, never backward.
• Output is inherently sequential. Token 50 can't be computed until token 49 exists. This is the fundamental reason long responses take time, and it shapes every optimization in post #7. Reading the prompt can be done in parallel; writing the answer cannot.

This also reveals two distinct phases, which the rest of the series will lean on:

   ┌──────────────┐     ┌────────────────────────────┐
   │   PREFILL    │ ──► │        DECODE              │
   │ read prompt  │     │ generate tokens one by one │
   │ all at once  │     │ sequential, one per step   │
   │ (fast/parallel)    │ (slow part — the bottleneck)│
   └──────────────┘     └────────────────────────────┘

The crucial choice: how to pick the next token

Here's the part most people don't realize. The model doesn't output a word — it outputs a probability distribution over every possible next token (post #4). Something has to decide which one to actually pick. That decision is sampling, and it's where a lot of the model's apparent personality comes from.

Say the distribution after "The weather today is" looks like:

   "sunny"   45%   ████████████████████
   "cloudy"  25%   ███████████
   "warm"    15%   ███████
   "rainy"   10%   ████
   "cold"     5%   ██

What now?

Option A — always take the most likely (greedy). Pick "sunny" every time. Deterministic and safe, but it makes the model repetitive and bland, and it can get stuck in loops ("the the the"). Great for tasks with one right answer, dull for writing.

Option B — sample proportionally. Roll a weighted die: 45% of the time "sunny," 25% "cloudy," and so on. Now the model is varied and creative — and non-deterministic, which is why asking the same question twice can give different answers.

In practice we tune how adventurous that die is, with a few knobs:

The knobs you can actually turn

Temperature rescales the distribution before sampling — it's the single most important knob.

   low temperature (≈0.2):  sharpen toward the top choice
        "sunny" 80%  "cloudy" 12%  "warm" 5%  ...     → focused, predictable

   high temperature (≈1.2): flatten the distribution
        "sunny" 32%  "cloudy" 24%  "warm" 20% ...      → diverse, surprising, riskier

• Low temperature → focused, consistent, repetitive. Use it for code, math, data extraction, anything with a correct answer.
• High temperature → creative, varied, occasionally unhinged. Use it for brainstorming, fiction, lots-of-options tasks.

Top-p (nucleus sampling) is a popular companion: instead of considering all tokens, keep only the smallest set whose probabilities add up to, say, 90%, and sample from those. It cuts off the long tail of absurd options while keeping healthy variety. (A relative, top-k, just keeps the k most likely tokens.)

Together these let you dial the model from "boringly reliable" to "wildly inventive." Picking well for the task is half of getting good results.

Where the context window bites

We met the context window in post #3 as a token budget. Inference is where it actually bites. At every single decode step, the model attends over all prior tokens — prompt plus everything generated so far. Two implications:

• The window has to hold the prompt and the growing response. A long prompt leaves less room to answer.
• When a conversation exceeds the window, something has to give: the oldest messages get dropped, summarized, or the request is refused. That "amnesia" in long chats isn't the model forgetting — it's the harness pruning to fit (post #10).

And recall from post #4 that attention cost grows roughly with the square of the sequence length. So a long context isn't just a memory question; it's a real compute cost paid on every token. That tension is exactly what the inference engine is built to manage.

"Streaming" and why it feels alive

When you watch a model type its answer word by word, that's not a UI gimmick — it's the decode loop made visible. Each token is sent to your screen the instant it's produced, rather than waiting for the whole response. Because decode is sequential anyway, streaming is essentially free, and it makes the wait feel much shorter. The first token typically takes the longest (that's the prefill phase finishing); after that, tokens stream at a fairly steady pace.

Putting it together

A single inference request looks like this end to end:

   your text
      │  tokenize (post #3)
      ▼
   [tokens] ──► PREFILL: model reads the whole prompt at once
      │
      ▼
   DECODE loop:
      run model ─► probability distribution ─► sample (temperature/top-p)
        ▲                                            │
        └────────── append token, repeat ◄───────────┘
      │  until a stop token or length limit
      ▼
   [output tokens] ──► detokenize ──► text on your screen (streamed)

Every part of this you can now name: the tokenizer (post #3), the transformer doing the prediction (post #4), the weights shaped by training (post #5), and the sampling choices we just covered.

Which raises the question every company serving an LLM has to answer: how do you run this loop for millions of users at once without going broke? That's the job of the inference engine.