Remember learning to add?
Every chatbot started there. This is the whole story, from counting on your fingers to ChatGPT, one scene at a time. Scroll to begin.
Three apples and four apples
You counted them all. Seven. Then you learned to do it without the apples. A number can stand for a pile of things.
It was never wrong. It had no idea what an apple was.
A computer is a very fast calculator.
Every chatbot is still one underneath. The whole story is about turning words into numbers so a calculator can work with them.
A letter for a number you don't know yet
If 2x + 1 = 7, what is x? Balance both sides until they match.
A language model is full of unknowns like x, called parameters. The one at the end of this story has about 82 million.
A point is two numbers. Distance is a triangle.
Drag the point
Later, every word becomes a point. Similar words sit close together.
Walk around a circle and you draw a wave
Waves like this will come back. They tell a model the order of your words.
The slope tells you which way is downhill
The curve is error. Stepping downhill on it is gradient descent, and it is how every language model learns.
One table of numbers moves every arrow at once
A list of numbers is a vector. A table is a matrix. Language models are mostly this multiplication, billions of times, on graphics chips.
One roll is a surprise. A thousand rolls are a pattern.
A die doesn't have to be fair. A loaded die lands on some faces more often. Hold on to that image.
Learn the formula from examples
Each dot is an apartment. The model is a line: rent = w × size + b. Train it and watch the line find the dots.
Your model, two parameters strong
It maps a number to a number. Now the question everything else answers:
What if the input were words, and the output were the next word?
A die with a word on every face
Which word comes after "the"? Some are far more likely. The words so far decide how the die is loaded. This one was loaded by counting Alice in Wonderland.
Load the die. Roll. Add the word. Repeat.
That is the entire job of a language model. The rest of the story is people finding better ways to load the die.
Part two
How we got here
For sixty years nobody said "large language model." It was called natural language processing.
Claude Shannon counts English
Shannon treated English as probabilities and built fake English by picking each word based on the one before. The first loaded word die.
Ignoring context:
Using the word before:
A Mathematical Theory of Communication, Shannon, 1948
Can a machine talk like us?
Alan Turing asked whether a machine could chat so well you couldn't tell it from a person: the imitation game. In 1956, a summer workshop at Dartmouth gave the field a name, artificial intelligence.
Computing Machinery and Intelligence, Turing, 1950
ELIZA, the first chatbot
Joseph Weizenbaum's program matched patterns and filled in canned replies. It understood nothing. People confided in it anyway.
ELIZA, Weizenbaum, 1966
Rules break the moment someone says something nobody planned for.
In language, that is constantly. Hand-written rules kept hitting that wall, and funding dried up in the AI winters.
Stop writing rules. Start counting.
Count which words follow which in huge piles of text, and load the die from the counts. An n-gram model. This one learned from Alice. Try a word Alice never used.
An unseen word has a count of zero: sparsity. Counting had hit its limit.
Words become points
Neural networks gave every word a list of numbers, an embedding, and learned those numbers. Words used alike land close together, so knowing "cat" helps with "kitten." Your geometry homework, back again.
Reading one word at a time, with a fading memory
Recurrent networks read in order and carry a memory forward. By the end of a long sentence, the start is a blur. LSTMs added gates to remember longer.
Long Short-Term Memory, Hochreiter & Schmidhuber, 1997
The bottleneck
Translators read a whole sentence into one single vector, then wrote the translation from it. That is a lot to squeeze through one list of numbers.
Sequence to Sequence Learning, Sutskever et al., 2014
Attention: look back at every word
Let the model look back at every earlier word and decide how much each one matters right now. The bottleneck was gone.
Who does "it" look at?
Example weights, made by hand. A real model computes these in every layer.Attention is all you need
The Transformer dropped one-word-at-a-time reading. Every word looks back at every earlier word, all at once. Each word sends a query (what am I looking for?), offers a key (what do I contain?) and a value (what can I pass along?). Gray squares are the future, hidden by the causal mask.
Attention Is All You Need, Vaswani et al., 2017
"Dog bites man" and "man bites dog" have the same words
Attention alone can't tell them apart. So every position gets its own pattern, built from sine waves of different speeds. Trigonometry, back again.
One block, stacked again and again
Wake a real one
DistilGPT-2: a smaller student of GPT-2, 82 million parameters, 6 blocks, 12 attention heads each. About 85 MB, running on your device. Nothing you type leaves it.
Downloads once. Your browser keeps it for next time.
The distillation method, Sanh et al., 2019
ChatGPT
The same next-word machine, plus two more rounds of training. Instruction tuning: examples of good answers. RLHF: people ranked answers, and the model learned their taste. Then a chat window. A million users in five days.
What each scene became
- CountingYour words become token IDs, plain numbers.
- AlgebraThe unknowns are parameters: billions of them.
- Geometry, trigTokens become points. Waves mark order.
- CalculusGradient descent tuned every parameter.
- Linear algebraEvery block is matrix multiplication.
- ProbabilityThe output is a loaded die.
- Shannon, n-gramsThe job: predict the next token.
- ELIZAThe chat window, and why rules weren't enough.
- Word vectorsEmbeddings.
- RNNsThe memory problem attention solved.
- TransformersThe engine that loads the die.
- Scale + RLHFFluent writing that answers you.
Watch it think
This is DistilGPT-2 from 2019, without ChatGPT's extra training. It doesn't answer you. It continues you. Turn on the chat format to coax answers out of it the old way.
What it did with your last message
Send a message. Every step shows up here.
1. Tokens and their ID numbers (· is a space)
2. Embeddings.
3. Six blocks of 12 attention heads and an MLP.
4. The loaded die: scores for all 50,257 tokens, top faces:
5. Roll, append, repeat:
Real probabilities, computed in your browser. The browser version of this model doesn't expose its attention weights, so those aren't shown.
A calculator, loading a die, one token at a time.
Every chatbot you use does this, with more parameters and more training.