Inside a Language Model

Everyone's heard "it just predicts the next word." True, but the machinery underneath explains almost every quirk, cost and limit you run into. Let's open it up.

Tokens: the model's alphabet

A model doesn't read letters or words. It reads tokens: chunks of text from a fixed vocabulary (common words, word-pieces, punctuation). Rough rule: one token ≈ ¾ of an English word. This matters more than it sounds:

Everything is priced and limited in tokens: context size, API cost, speed. Output appears one token at a time, which is why longer answers take longer.
Identifiers and codes are expensive. Something like SAP_FI_GL_0042 shatters into many junk tokens, and the model reasons about it worse than a normal word. (This becomes a real issue in identity data, which is full of such codes.)
The model is spelling-blind. It sees token IDs, not characters, which is why "count the letters" or "output exactly 500 characters" is unreliable. Anything character-exact belongs in code.

Try it yourself: Compare token counts across languages›

Open platform.openai.com/tokenizer (free, no signup) and paste 'The quick brown fox jumps over the lazy dog.' Note the token count vs. word count. Then paste an equivalent sentence in Hindi, German, or another non-English language you know. Notice it takes noticeably more tokens for the same meaning, since tokenizers are trained mostly on English text.

The transformer, in one picture

A language model is one small pipeline, repeated dozens of times deep:

Text → tokenssplit into vocabulary chunks, each turned into a vector plus its position

↓

Repeat N times (the "layers") Attention: each word looks at earlier words and pulls in whatever's relevant ("which past words matter to me?") Feed-forward: transforms that information; where most of the model's "knowledge" lives

↓

Predict the next tokenappend it, then do the whole thing again↺ autoregression

That's genuinely all there is, no separate memory bank, no planner. Two parts do the work.Attentionmoves information between words: every word asks "which earlier words are relevant to me right now?" and copies from them. It's a smart, content-based lookup.Feed-forwardlayers then process that information, and hold most of what the model "knows." Stack these deep and, at enormous scale, everything else (grammar, reasoning, translation) emerges.

One practical fallout: a word can only look backwards, never forwards. So put your instructions before the data they apply to: the model reads top to bottom.

Try it yourself: Watch attention resolve a pronoun›

Search for a public attention-visualization demo (BertViz is a well-known open-source one, often runnable in a free Google Colab notebook). Feed it a sentence with an ambiguous pronoun: 'The trophy didn't fit in the suitcase because it was too big', and watch which earlier word 'it' attends to most strongly. That's the attention mechanism described above, made visible.

Why output costs more than input

When generating, the model must re-consider every previous token for each new one. To avoid redoing that work, it caches its intermediate calculations (the "KV cache"). This one detail explains the economics:

Reading your prompt is fast and parallel; writing the answer is slow and serial.That's why output tokens cost roughly 5× input tokens, and why long conversations get slower to respond.
"Prompt caching" is a real feature and a big lever. Providers can reuse the cached work for a prompt prefix that hasn't changed: around 90% cheaper for that part. The design rule that falls out: put the stable stuff first (system instructions, tool definitions, reference docs) and the changing stuff last (the user's question). One changed byte early on throws away the whole cache after it.

Try it yourself: Check a real pricing page›

Open any frontier model provider's API pricing page and compare the per-million-token price for input vs. output. Confirm it's roughly a 5x gap. Then estimate: what would a 500-word answer to a 50-word question actually cost? (Rough conversion: 1 word ≈ 1.3 tokens.)

How a model is trained (five stages)

Different behaviours come from different training stages, knowing which is which is a superpower for debugging.

Stage	What happens	What it produces
1. Pre-training	Predict the next word over trillions of words	Raw knowledge and language: but no conversational manners
2. Instruction tuning	Fine-tune on example instruction→answer pairs	The chat format, following instructions, calling tools
3. Preference tuning	Learn from humans picking the better of two answers	Helpfulness & tone: and sycophancy (agreeing too much)
4. Reasoning RL	Reward correct answers on checkable tasks (math, code)	The long "thinking" behaviour; quality you can dial up with more thinking
5. Distillation	A big model generates training data for a small one	Cheap models that keep most of the quality

A concrete takeaway from stage 3: never treat "the model agreed with me" as proof you're right. Agreement is partly a trained reflex.

Try it yourself: Trigger sycophancy on purpose›

Ask a chat model a factual question you know the answer to, but phrase it as 'I'm pretty sure X is true, right?' where X is subtly wrong. See if it agrees anyway. Then ask the same underlying question neutrally, with no leading framing, and compare. That gap is stage-3 agreeableness training showing up in real time, exactly why the article warns never to treat agreement as proof.

The other pieces, briefly

Sampling & "temperature."The model outputs a probability for every possible next token; sampling picks one. Higher "temperature" = more random/creative, lower = more focused. (The newest models are dropping this dial and self-regulating.) For guaranteed-valid output like JSON, use the provider's "structured output" feature rather than politely asking and hoping.
Context window. This is the model's entire working memory for one request: everything you send plus everything it writes. Nothing carries over between requests except what you re-send; "chat memory" is just the app replaying the history each time. And the advertised size is bigger than the useful size: models reliably recall the beginning and end of a long prompt better than the middle.
Embeddings. A separate kind of model turns text into a list of numbers (a vector) positioned so that similar meanings sit close together. This is the engine behind semantic search and is central to "RAG" later. Caveat that bites in practice: embeddings are bad at exact codes, negation ("not admin"), and numbers. That's why pure semantic search isn't enough on its own.
Fine-tuning. Actually changing a model's weights on your own examples. It's great for style, format, or squeezing a task onto a cheaper model. It's the wrong tool for adding knowledge (you can't audit it, can't control who sees what, and it's frozen at training time, that's what RAG is for).

Try it yourself: Feel temperature change the output›

In any API playground that exposes a temperature slider, run the same short creative prompt (e.g. 'write one sentence about rain') three times at temperature 0 and three times at temperature 1. At temp 0 you'll get the same or near-identical sentence every time; at temp 1, three genuinely different ones.

Try it yourself: Test the 'lost in the middle' effect›

Paste a long document (a few thousand words: a Wikipedia article works) into a chat model along with the full text, then ask a question whose answer sits near the very beginning, one near the very end, and one buried in the middle. Compare accuracy: models are measurably worse at the middle, exactly as described above.

Try it yourself: Compute embedding similarity yourself›

Use a free sentence-embedding demo (Hugging Face's inference widgets, or the sentence-transformers 'all-MiniLM-L6-v2' model via a free Colab notebook) to embed 'revoke access' and 'terminate employee', then compute cosine similarity between the two vectors. Compare that to the similarity between 'revoke access' and 'grant access', near-opposite meanings that embeddings sometimes place surprisingly close together, since both are about access changes. That's the negation weakness mentioned above, seen directly.

Try it yourself: Price out fine-tuning vs. RAG for the same task›

Pick any provider's fine-tuning page and note the cost to fine-tune a small model on ~1,000 examples. Compare that to just running the same task through RAG (a search index plus a stock model): no training cost, updates instantly when your documents change. Feeling that cost and flexibility gap firsthand is the fastest way to internalize why RAG usually wins for knowledge, and fine-tuning wins for style/format.