Stop Pretending You Understand LLM Terms Without a Map

Last month, I was reviewing a pull request where someone had implemented token streaming for our chatbot API. The comment in the code said "reduces latency by 40%." I nodded and approved it, but honestly? I had no idea why tokens specifically mattered, or what a KV cache even was supposed to do. I just knew it worked and that I was supposed to know better.

That's the moment I realized I've been building LLM features like someone following a recipe in a foreign language, making it work without understanding why the ingredients matter or how they interact. When you're shipping production features, this gap doesn't hurt until it does. Until you're debugging a quantization issue at 2 AM, or trying to explain to your team why switching from standard attention to GQA might save us 30% on inference costs.

I found a glossary that changed this for me, and I want to tell you why it's different from every other "LLM terms explained" thing I've read.

The Problem with Alphabetical Understanding

Every glossary I've read before treats LLM concepts like a dictionary-"attention" entry, "softmax" entry, "logits" entry. You end up chasing rabbit holes. The article I just read flipped this entirely: dependency order instead of alphabetical order. This seems small until you realize it's actually how understanding actually works in your brain.

I think in sequences. As a sysadmin-turned-developer, I understand caches, control loops, and error signals. The glossary I found leverages that, it explains tokens first, then embeddings as the bridge from tokens to math, then matrix operations, then the transformations that build on top of those primitives. By the time you hit "Mixture of Experts," you already have the mental model.

Tokens: Where Text Becomes Numbers

Here's what actually clicked for me: a token isn't a word. It's a compression unit, like an LZ algorithm learned from real data. "Computing" might be one token. "Quadcopter" becomes "quad" + "cop" + "ter". The tokenizer is just a fancy lookup table, token 4521 equals " cat".

Why this matters in production: your token limit isn't a word limit. A prompt that looks short in English might be 1000 tokens. I built a feature that truncated based on word count instead of token count, and it failed silently until our quality metrics dropped. The tokenizer is deterministic but not intuitive, you have to respect it.

The Foundation: Embeddings and Matrix Math

Embeddings are the bridge I was missing. A 640-dimensional vector represents each token. These vectors organize themselves during training by meaning-"cat" and "feline" end up close in vector space. If you've used vector databases for semantic search (I have), you already understand this. It's the same concept, just internal to the model.

Everything that follows is matrix multiplication with non-linearity layered in. A "linear layer" is just y = W @ x. Parameters are the numbers in those matrices. A 180M parameter model has 180 million floats that get adjusted during training. The architecture is the circuit board; training solders in the component values.

This is profound because it means: an LLM isn't magic. It's a giant function composed of hundreds of matrix multiplications with activation functions between them. Understanding that changed how I think about quantization, pruning, and inference optimization.

My Take: Honesty About What We Don't Know

What I respected most about this glossary is that the author admits they don't fully master everything in it. They learned it by asking an LLM to explain it, slowly, with analogies that stuck. And they built it as a reference notebook, something to come back to when you hit a term in production and need to understand it quickly.

This is how I actually work now. I maintain my own notebook of these glossary entries, with examples specific to our models. When someone says "MoE would save us tokens per inference," I can actually trace through what that means: a Mixture of Experts model routes different inputs through different experts, reducing computation per token. It's not magic anymore.

What I'm Still Chewing On

I'm still solidifying my intuition around KV caches and why the √d factor matters in attention. These aren't things you can deploy wrong in the same way, they're optimizations and design patterns. But understanding them means understanding where latency actually comes from in inference.

The gap between "making it work" and "understanding why it works" used to feel acceptable. Now, working on features that matter, it doesn't.

What's one LLM term that's been living in your codebase that you've never fully understood?

Source: This post was inspired by "From "token" to "MoE": the LLM glossary in dependency order" by Dev.to. Read the original article