I spent three hours debugging why Claude was giving inconsistent responses to structurally identical prompts last month. Same input, different outputs. My first instinct? Blame randomness. My second instinct? Blame the API. My actual problem? I had no mental model for what was actually happening inside the model, so I couldn't reason about it properly.
That's when I realized I'd been using LLMs in production for over a year without actually understanding the mechanics. I could prompt-engineer circles around most people, but I couldn't explain why certain phrasings worked better than others. I was pattern-matching, not thinking. Reading through how these models actually work forced me to confront how much I was just cargo-culting with AI.
Breaking Down the Pipeline: Tokens to Text
Here's the thing that clicked for me: LLMs don't think in words. They think in numbers. The moment you hit enter on your prompt, the text gets shredded into tokens, and this matters more than most developers realize.
A tokenizer doesn't split on word boundaries like you'd expect. "Building" might be one token or three, depending on the model's vocabulary. Whitespace is a token. Punctuation is a token. When I finally understood this, a whole class of weird LLM behaviors suddenly made sense to me. That's why markdown formatting sometimes helps, you're not just making the prompt prettier for a human reviewer; you're literally restructuring the token sequence in ways that influence how the model processes information.
Each token becomes a vector, a point in a high-dimensional space, and here's where it gets interesting: these embeddings aren't hand-crafted by engineers. They're learned during training. The number 1975 doesn't mean anything on its own, but in the context of millions of token relationships, it encodes something meaningful about the word "build."
Self-Attention: The Actual Intelligence
The transformer architecture's real genius is self-attention. Without it, the model would process each token in isolation, which is useless for language. With it, each token can look around and ask: "Which other tokens should influence how I'm understood right now?"
This is what makes context actually matter. The word "build" in "the build failed" has a different representation than "build" in "let's build a system" because attention mechanisms have wired in the relationships between tokens based on what it learned during training.
I've started thinking about this when I write prompts. Complex instructions need space. If you jam everything into one dense paragraph, you're making the attention mechanism work harder to figure out which parts of your prompt relate to which parts of your question. Breaking things into numbered lists or separate paragraphs isn't just good human readability, it's helping the model's attention heads find meaningful relationships.
The Path From Context to Prediction
After tokens flow through multiple transformer layers, the model generates logits, raw scores for what token should come next. These aren't probabilities yet; they're scores that get converted into probabilities using softmax. A higher logit means "more likely next token."
Then decoding strategy kicks in. Greedy selection picks the highest probability. Temperature adjusts how "confident" or "random" the selection is. Top-k and top-p sampling introduce controlled randomness. This is the part that's actually user-controllable, and most developers don't realize they can tweak it. When I set temperature to 0, I'm saying "be deterministic." At 1.0, I'm saying "be creative." This explains why my "inconsistent responses" problem existed, I was using different temperature settings across different API calls without realizing it.
My Take: What This Changes
Understanding this pipeline won't make you a better prompt engineer overnight, but it reframes how you think about LLMs. They're not search engines. They're not stored knowledge bases. They're fundamentally predicting the next token based on statistical patterns learned from training data.
This means:
- LLMs struggle with things that require exact knowledge they weren't trained on
- Context window size matters because attention has to process all tokens
- Consistent formatting in prompts reduces noise in the token sequence
- Reproducibility requires controlling decoding parameters
The gap between understanding "LLMs predict tokens" and actually reasoning about LLM behavior is where most developers fail. We need this mental model.
What's Your Model?
When you next use an LLM, try this: before you hit submit, think about how your prompt will tokenize. Will markdown help? Is your context organized in a way that makes attention relationships obvious? Are you controlling for randomness when you need consistency?
What aspects of LLM behavior have confused you? Now you have a framework to reason about them.
Source: This post was inspired by "How Does LLM Actually Work? From Prompt to Prediction" by Dev.to. Read the original article
