What Is a Large Language Model?

In post #1 I promised that one idea carries most of the weight:

A large language model predicts what text comes next.

Now let's take that seriously, because once you really believe it, a lot of mysterious behavior suddenly makes sense — including the model's failures.

The autocomplete that ate the world

You already use a tiny language model every day. Your phone's keyboard suggests the next word as you type. Type "I'll be there in five" and it offers "minutes." That keyboard is doing exactly what a large language model does, just with a laughably small brain and a short memory.

A large language model is the same idea scaled up by a factor of millions:

• Instead of looking at the last word or two, it looks at thousands of words of context.
• Instead of a few simple rules, it has hundreds of billions of tunable numbers ("parameters") that encode patterns learned from a huge slice of the internet, books, and code.
• Instead of suggesting one likely word, it produces a full probability distribution over every possible next token, then picks from it.

Here's the loop, in pseudocode:

text = "The capital of France is"
while not done:
    probabilities = model(text)        # a score for every possible next token
    next_token = pick_from(probabilities)
    text = text + next_token           # append and repeat

Run that loop and you get:

"The capital of France is"      → " Paris"
"The capital of France is Paris" → "."
...

This is called autoregression: each new token is fed back in to help predict the next one. The model writes by reading what it has written so far. We'll come back to this loop again and again — it's the beating heart of inference (post #6).

Why next-token prediction is secretly profound

"Okay," you might say, "but autocomplete isn't intelligence." Here's the leap that surprised the whole field.

To predict the next word really well, across all of human writing, you are forced to learn an enormous amount about the world. Consider what it takes to finish these:

• "The opposite of hot is ___" → you need a concept of antonyms.
• "2 + 2 = ___" → you need arithmetic.
• "She poured water into the cup until it was ___" → you need physical intuition.
• "The detective realized the butler was lying because ___" → you need a model of motivation and plot.
• "def factorial(n): return ___" → you need to understand code.

There is no separate "knowledge module" the model consults. The only objective it was ever trained on is "predict the next token," and yet, to do that well across billions of examples, it had to absorb grammar, facts, reasoning patterns, coding conventions, and a rough physics of everyday life. Capability emerged as a side effect of getting really good at autocomplete. That's the whole surprise of the modern era of AI in one sentence.

What "large" actually means

The "large" in LLM refers mostly to two numbers:

1. Parameters — the tunable weights inside the network. Modern frontier models have hundreds of billions to trillions of them. Each one is just a number; collectively they store everything the model "knows."
2. Training data — the amount of text the model learned from, measured in tokens. Frontier models train on trillions of tokens: a meaningful fraction of the high-quality public text that exists.

A rough intuition: parameters are the size of the brain, training tokens are the amount of experience. Both matter, and there's a sweet spot between them — a famous line of research (the "scaling laws") showed that for a given compute budget there's an optimal balance, and that for a long time models were actually too big and under-trained.

The eerie thing about that curve is how smooth and predictable it has been. For years, pouring in more scale reliably bought more capability. That predictability is a big part of why so much money flowed into the field.

What the model is not

This is the most useful section in the post, because misunderstanding it is the source of nearly every "AI is dumb" and "AI is magic" hot take.

• It is not a database. It cannot quote a source it saw once. It compressed the patterns of its training data into weights, losing the originals. Ask it for an exact quote or an obscure citation and it may confabulate — generate something that has the right shape but is false. This isn't lying; it's the model doing its only job (producing plausible next tokens) in a situation where plausible and true have come apart.
• It has no memory between conversations by default. Each request is, to the raw model, a blank slate plus whatever text you put in front of it right now. Everything that feels like memory is the harness re-feeding earlier text (posts #10 and #14).
• It doesn't know what it doesn't know. Confidence in the output is not calibrated to truth. A model can be equally fluent whether it's right or hallucinating. Managing this is a central theme of the back half of the series.
• It doesn't run code or browse the web on its own. A bare model only emits text. Connecting it to the world is something we bolt on (post #11).

Hold these close. Most "gotcha" failures you'll see online are just the model behaving exactly as a next-token predictor should, used in a situation that demanded something else.

A useful mental image

Picture a staggeringly well-read improviser. They've absorbed a huge fraction of everything ever written, but they have no notes in front of them and no way to check anything. You hand them a script that cuts off mid-sentence, and their job is to continue it in the most fitting way possible, instantly, one word at a time, never looking back to edit.

That improviser is astonishing at fluent, on-pattern continuation and genuinely weak at precise recall, arithmetic done silently in their head, or admitting "I don't know." Almost everything we build in the rest of this series — tools, retrieval, harnesses, agents — exists to compensate for the improviser's weaknesses while exploiting their strengths.

Next, we get concrete about the very first step in that pipeline: how your text gets chopped into the tokens the model actually sees. It's weirder, and more consequential, than you'd expect.