Making Models Faster & Cheaper

Post #7 made the serving efficient. This post makes the model itself cheaper to run. These techniques are why a capability that cost a fortune two years ago might now run on a laptop — and why the price of a given level of intelligence keeps falling off a cliff.

Four big ideas: quantization, distillation, speculative decoding, and mixture-of-experts. Each attacks cost from a different angle.

Quantization: use fewer bits per number

Recall that a model is billions of numbers (post #2). By default each is stored in fairly high precision — say 16 bits each. Quantization asks: do we really need that many bits?

The analogy: think of an image. A photo at full color depth looks great but is huge. Drop it to fewer colors and the file shrinks dramatically while looking nearly identical. Quantization does the same to a model's weights — store each number in 8 bits, or 4, or even fewer.

Why it's such a big deal, given post #7's lesson that memory bandwidth is the bottleneck:

• Smaller model = less memory traffic = faster. If decode speed is limited by how fast you can stream weights from memory, halving the weights nearly doubles the speed.
• Smaller model = fits on cheaper hardware. A model that needed a data-center GPU might now fit on a consumer card — or your phone.
• Smaller model = more room for the KV cache and bigger batches (post #7), which compounds the throughput win.

The catch is accuracy. Round the numbers too aggressively and the model gets dumber or starts making errors. The art is quantizing the parts that tolerate it while protecting the sensitive ones. Modern 8-bit and even 4-bit methods are good enough that the quality loss is often barely noticeable — which is why quantized models are now everywhere, especially for running locally.

Distillation: train a small model to imitate a big one

Quantization shrinks a model's precision. Distillation shrinks its size by training a small "student" model to mimic a large "teacher."

The insight is that the teacher's full output distribution is far richer than a plain right/wrong label. When the teacher says "the answer is 70% 'sunny', 25% 'cloudy'," the student learns the teacher's nuance and uncertainty, not just the final pick. A small model trained this way can capture a remarkable fraction of a much larger model's ability on the tasks it was distilled for.

This is a major reason the small, fast, cheap models you can run today are so good: many of them learned from much larger, more expensive siblings. The trade-off is breadth — a distilled student tends to be excellent in the domains it was trained on and weaker outside them.

Speculative decoding: guess ahead, verify in bulk

This one is clever, and it directly attacks the sequential bottleneck from post #6 — that you can only make one token at a time.

The trick uses two models: a small fast draft model and the big accurate target model.

   1. The small model quickly drafts several tokens ahead (a guess):
         "The cat sat on the"  →  draft: " mat and looked"

   2. The big model checks ALL of those guesses in a SINGLE pass
      (verifying is parallel; generating was not):
         " mat"    ✓  accept
         " and"    ✓  accept
         " looked" ✗  reject → big model supplies the correct token instead

   3. Keep the accepted tokens, continue from the correction.

Why it's a free lunch (almost): verifying several tokens at once costs the big model roughly the same as generating one token, because — per post #7 — the expensive part is loading the weights, not the math. So every accepted guess is a token you got nearly for free. When the small model guesses well (and on easy, predictable text it often does), you get a big speedup with zero quality loss — the output is exactly what the big model would have produced alone, because the big model has final say on every token.

Mixture-of-Experts: don't use the whole brain for every token

The last idea changes the model's architecture. A normal ("dense") model uses all of its parameters to process every token. A Mixture-of-Experts (MoE) model instead contains many specialized sub-networks ("experts") and, for each token, a small router activates only a few of them.

   Dense model:   every token  ──►  ALL parameters fire        (expensive)

   MoE model:     every token  ──►  router picks 2 of 64 experts
                                    ──► only those fire          (cheap per token)

The win: a model can have a huge total number of parameters (lots of stored knowledge and capacity) while only using a small fraction for any given token (cheap to run). You get much of the capability of a giant model at a fraction of the per-token compute. The cost is complexity and memory — all those experts still have to be stored, even if only a few fire at a time — plus the engineering of keeping the routing balanced. Many frontier models today are MoE under the hood.

How they stack up

   technique            shrinks…           main win              main cost
   ──────────────────────────────────────────────────────────────────────────
   quantization         bits per weight    speed + fits hardware  slight accuracy
   distillation         number of params   small capable models   narrower breadth
   speculative decode   wall-clock time    faster, same output    needs a draft model
   mixture-of-experts   compute per token  big capacity, cheap    storage + complexity

These aren't either/or — a deployed model often uses several at once: an MoE model, quantized to 8 bits, served with speculative decoding, possibly distilled from a larger teacher. Layered together, they're the engine behind the relentless drop in the cost of intelligence.

Why this matters for everyone else

• Prices keep falling. The cost per token for a given capability level has been dropping dramatically year over year, and these techniques are a big reason. Plan for "good enough" to get cheaper fast.
• Small models are punching up. Don't assume you need the biggest model. A distilled, quantized small model is often plenty for classification, extraction, routing, and many everyday tasks — at a tiny fraction of the cost and latency.
• Local and on-device AI is real. Quantization especially has made it practical to run genuinely capable models on laptops and phones, with the privacy and offline benefits that brings.

The takeaway

That's the end of the "how the machine works" half of the series. From here we go up the stack — to how we actually talk to these models and wrap them into useful products. It starts with the deceptively deep art of the prompt.