Tokens & Tokenization

We keep saying a model predicts "the next token." Time to find out what a token actually is — because tokens quietly explain a surprising number of an LLM's quirks, from why it's bad at spelling to why your API bill looks the way it does.

Models don't read letters or words

A neural network only does math. It can't add "cat" and "dog"; it can only add numbers. So before any text reaches the model, it has to be converted into numbers. That conversion is tokenization.

The naïve approaches both fail:

• One number per letter? Too granular. The model would waste its capacity rediscovering that "t-h-e" is a common pattern, and sequences would get enormously long.
• One number per word? Too coarse. There are millions of words, plus misspellings, names, code, emoji, and other languages. You'd need an impossibly huge vocabulary and still choke on anything new.

So modern models use a middle path: subword tokens. Common words get their own token; rarer words get split into pieces. The vocabulary — the full set of tokens a model knows — is typically somewhere around 50,000 to 200,000 entries.

What tokens actually look like

Here's how a sentence might break apart (using · to mark a leading space):

A few things to notice, because each one matters in practice:

• The leading space is part of the token. "·is" (with a space) is a different token from "is". This is why models are so sensitive to whitespace.
• Common words are single tokens; rare words shatter. "Tokenization" became "Token" + "ization." A made-up or unusual word might become five or six pieces.
• A rough rule of thumb in English: ~1 token ≈ 4 characters ≈ ¾ of a word. So 1,000 tokens is roughly 750 words. Other languages, code, and math can be far less efficient.

Why this is the cause of so many quirks

Once you know text is tokens, a whole category of "why is the AI so dumb about this?" moments dissolves.

Why models miscount letters. Ask a model how many "r"s are in "strawberry" and it has historically struggled. Why? It never saw the letters s-t-r-a-w... It saw something like ["str", "aw", "berry"]. Asking it to count letters is like asking you to count the brushstrokes in a word you only ever saw as a printed logo. The information has been partly abstracted away.

Why rhyming and wordplay can be shaky. Sound and spelling live at the character level, but the model operates on chunks. It can still do these things because it learned patterns about tokens, but it's working with one hand tied.

Why some languages cost more. Tokenizers are usually trained mostly on English text, so English is tokenized very efficiently. A language with a different script may need two or three times as many tokens to say the same thing — which means it's more expensive to process and eats more of the context window for the same content.

Why formatting can break things. Numbers, code indentation, and unusual Unicode can tokenize in awkward ways, occasionally tripping up otherwise capable models.

Tokens are the unit of money and memory

Two of the most practical facts about working with LLMs come straight from tokens:

1. You pay per token. Providers bill by tokens in (your prompt) plus tokens out (the response). "Make the prompt shorter" is literally "spend less money." A verbose system prompt that runs on every request can dominate your bill.
2. The context window is measured in tokens. Every model has a maximum number of tokens it can consider at once — its context window. This includes your prompt and its answer. A "200K context" model can juggle roughly 150,000 words at a time. Run past it and the earliest text falls off the edge or the request is rejected. We'll see in post #6 why this limit exists and in post #14 how we work around it.

A surprising amount of practical "prompt engineering" and "context engineering" (post #9) is really just budgeting tokens well: deciding what's worth its space.

Embeddings: from token to meaning

One more step before the model proper. A token ID like 1842 is just an index — it carries no meaning by itself. So each token ID is mapped to a long list of numbers called an embedding — typically several thousand numbers, called a vector.

The magic is that these vectors are arranged so that meaning becomes geometry. Tokens with similar meanings end up near each other in this high-dimensional space. The classic illustration:

vector("king") − vector("man") + vector("woman") ≈ vector("queen")

Relationships between concepts show up as consistent directions in the space. "Male → female," "country → capital," "present → past tense" — each is roughly a constant nudge in some direction. The model isn't told any of this; it falls out of learning to predict text well.

These embeddings are the actual input to the network. So the real pipeline is:

The takeaway

Tokenization feels like a boring preprocessing detail, but it's load-bearing:

• It's why models stumble on spelling, counting letters, and some languages.
• It's the unit you pay in and the unit your context window is measured in.
• It's the bridge from raw text to the vectors where meaning lives as geometry.

Now that we know how text becomes vectors, we can finally open the box and look at the engine that turns those vectors into predictions: the transformer.