Remember learning to add?

Three apples and four apples

You put three apples next to four apples and counted them all. Seven. Then you learned to do it without the apples. That idea, that a number can stand for a pile of things, is the first step toward everything else on this page.

Then somebody handed you a calculator. You pressed buttons, and it was never wrong. It also had no idea what an apple was. It only knew the numbers you gave it.

The chat on the side is a calculator right now. Type 12 * 7, then try typing a word and see what happens.

Hold on to that feeling. A computer is a very fast calculator. Everything you will see later, every chatbot, is still a calculator underneath. The whole story is about how people turned words into numbers so a calculator could work with them.

Algebra: a letter for a number you don't know yet

Then numbers got letters. If 2x + 1 = 7, what is x? You balance the two sides until they match. The letter stands for an unknown number, and your job is to find it.

Keep this one in your pocket. A language model is full of unknown numbers like x. They are called parameters, and the small model at the end of this page has about 82 million of them. Training is the process of finding good values for all of them at once.

Points, distances and triangles

Geometry gave numbers a place to live. A point on a grid is just two numbers: how far across, how far up. And the distance between two points comes from the triangle they make, with the Pythagorean theorem.

Later in this story, every word becomes a point. Words that mean similar things end up close together, and "how similar are these two words?" turns into "how far apart are these two points?" Your geometry homework was secretly about chatbots.

Angles and waves

Trigonometry turned angles into numbers. Walk a point around a circle and track its height, and you get a wave: the sine wave. It repeats forever, like sound, tides and seasons.

Two callbacks are coming. The angle between two arrows becomes a way to measure how similar two words are. And sine waves of different speeds become the way a model knows the order of the words you typed.

Which way is downhill?

Calculus is the math of change. The derivative tells you how steep a curve is at any point. If you are standing on a hill in the fog, the derivative tells you which way is down.

Here the curve is error: how wrong a guess is. Each step uses the slope to move a little downhill, so the error shrinks. That move is called gradient descent, and it is how every language model is trained. Same idea, with millions of hills at once.

Lists of numbers, and tables that move them

A list of numbers is a vector. You can draw it as an arrow. A table of numbers is a matrix, and multiplying by a matrix moves every arrow at once: it stretches, spins or squishes the whole grid.

This is the workhorse. Almost everything a language model does is multiplying lists of numbers by tables of numbers, billions of times. The graphics chips built for video games, GPUs, happen to be excellent at exactly this.

Probability: what usually happens

Roll a fair die once and you can't predict it. Roll it a thousand times and a pattern appears: each face shows up about one time in six. Probability is the math of that pattern. Statistics is the habit of collecting lots of examples and letting the pattern show itself.

The chat is a die now. Send anything and it rolls.

A die doesn't have to be fair. A loaded die lands on some faces more often than others. Keep that image. It is the most important object in this whole story.

Learning a formula from examples

Now put the classes together. Machine learning builds a formula from examples instead of writing it by hand. Here each dot is an apartment: its size and its rent. The formula is a line, rent = w × size + b. The two unknowns, w and b, start as bad guesses. Algebra names them. Calculus finds which way to nudge them. Statistics supplies the examples.

The chat is this model now. Type a size, like 650, and it predicts a rent. Train it first and the prediction gets better.

This model maps a number to a number. Here is the question the rest of the story answers: what if the input were words, and the output were the next word?

A die with words on it

Picture a loaded die with a word on every face. Which word comes after "the"? Some words are much more likely than others, so the die is heavily loaded. The words you have already read decide how it is loaded.

The chat is that die now. Send anything and it rolls a word.

That is the entire job of a language model: look at the words so far, load the die, roll it, add the word, and do it again. Everything that follows, from 1948 to ChatGPT, is the story of people finding better ways to load the die.

Claude Shannon counts English

Before computers could do much of anything, the mathematician Claude Shannon wrote "A Mathematical Theory of Communication." In it, he treated English as a source of probabilities. He built fake English by picking each word based on the word before it, and the result looked strangely like real writing. That is the first loaded word die.

The chat plays Shannon's game now. Type a few words and it continues from your last one.

Read the paper: A Mathematical Theory of Communication, Shannon, 1948.

Can a machine talk like us? Someone names it "AI"

In 1950, Alan Turing asked whether a machine could hold a written conversation so well that you couldn't tell it from a person. He called it the imitation game. Today people call it the Turing test.

In 1956, a group of researchers met at Dartmouth College for a summer workshop, and John McCarthy's proposal for it gave the field its name: artificial intelligence. The work of getting computers to handle human language got its own name too: natural language processing, or NLP. For the next sixty years, this was NLP. Nobody said "large language model."

Read the paper: Computing Machinery and Intelligence, Turing, 1950.

ELIZA, the first chatbot

At MIT, Joseph Weizenbaum wrote ELIZA. Its most famous script played a therapist. It had no idea what you meant. It matched patterns in your sentence and filled in canned replies: say "I am tired" and it asks how long you have been tired. People still poured their hearts out to it, which alarmed Weizenbaum.

The chat is an ELIZA-style program now. Type "I am worried about my exam." Then type something it has no rule for.

This is rule-based NLP: people write the rules by hand. It works until someone says something the rules didn't plan for, and in language that happens constantly. Through the 1970s and 1980s, hand-written rules and "expert systems" kept hitting that wall, and funding dried up in what people call the AI winters.

Read the paper: ELIZA, Weizenbaum, 1966.

Stop writing rules. Start counting.

Shannon's idea came back with computers big enough to use it. Instead of writing rules, count how often words follow each other in huge piles of text. An n-gram model looks at the last few words, n minus one of them, and loads the die from the counts. Speech recognition and machine translation ran on this for decades.

The chat is a counting model now, trained on Alice. Type "the white", then end a sentence with a word Alice never uses, like "I saw a spaceship."

You just found the problem. If a word pair never appeared in the counts, its probability is zero, even when the sentence is perfectly fine. That is called sparsity. Look at more words to be smarter, and almost every long word sequence has a count of zero. Look at fewer words, and you forget what the sentence was about. Counting alone had hit its limit.

Words become points

In 2003, Yoshua Bengio and colleagues trained a neural network to load the die instead of counting. The trick: give every word a list of numbers, a vector, and let the network learn those numbers. Words used in similar ways end up with similar numbers. So if "cat" was seen a lot and "kitten" only a little, the model can still guess sensibly about kittens. Sparsity solved.

In 2013, word2vec made these word vectors cheap to learn from billions of words. Remember geometry? Each word is now a point, and close means similar. These vectors are called embeddings.

The chat reads this map now. Type "tuition" and it shows the nearest neighbors.

Read the papers: A Neural Probabilistic Language Model, Bengio et al., 2003 · word2vec, Mikolov et al., 2013.

Reading one word at a time, with a fading memory

A fixed window of words is still a window. Recurrent neural networks read a sentence one word at a time and carry a memory forward, so in principle they can use everything before. In practice, the memory fades: by the end of a long sentence, the beginning is a blur. The LSTM, from 1997, added a separate long-term memory with gates that decide what to keep and what to forget, and by the early 2010s these networks ran speech recognition and translation.

The chat reads like an RNN now. Type a long sentence and watch which words it still remembers.

In 2014, sequence-to-sequence models translated by reading a whole sentence into one vector, then writing the translation from that single vector. That is a lot to squeeze into one list of numbers. People called it the bottleneck.

Read the papers: Long Short-Term Memory, Hochreiter & Schmidhuber, 1997 · Sequence to Sequence Learning, Sutskever et al., 2014.

Attention: look back at every word

In 2014, Dzmitry Bahdanau and colleagues let the translator look back at every word of the source sentence, and decide how much each one matters right now. That is attention. The bottleneck was gone.

In 2017, a team at Google asked: what if attention is all you need? Their Transformer dropped the one-word-at-a-time reading entirely. Every word looks at every other word, all at once, which is perfect for GPUs and their giant matrix multiplications.

How does a word decide where to look? Each word makes three lists of numbers. A query: what am I looking for? A key: what do I contain? A value: what can I pass along? When a query matches a key well, that word's value flows in. Several sets of these run side by side, called heads, each free to track a different kind of link.

The chat shows attention now. Type a sentence and see which words can look at which.

Two rules make it work for writing. First, when predicting the next word, a word may only look backward, never ahead. That is the causal mask: the gray squares in the chat. Second, attention on its own can't tell word order: "dog bites man" and "man bites dog" contain the same words. So each position gets its own pattern added in, and the original Transformer built those patterns from sine waves. Trigonometry, back again.

A Transformer stacks the same block many times. Inside each block, attention lets words gather information from each other, then a small network called the MLP works on each word by itself. Shortcut paths called residual connections carry each word's numbers past each step, and normalization keeps the numbers in a steady range.

Read the papers: Neural Machine Translation by Jointly Learning to Align and Translate, Bahdanau et al., 2014 · Attention Is All You Need, Vaswani et al., 2017.

Then everything got very, very big

OpenAI took the Transformer and played the oldest game on this page: guess the next word. They trained it on a huge amount of text first, with no labels needed, because the text itself supplies every answer. Generative, Pre-trained, Transformer: GPT. In 2018 came GPT-1. In 2019, GPT-2, which wrote surprisingly fluent paragraphs. In 2020, GPT-3, which could do new tasks from a few examples in the prompt. Around then, people started calling these large language models.

The idea never changed. Load the die, roll, append, repeat. What changed was how many parameters loaded it, and how much text they learned from.

You can run one of these. DistilGPT-2 is a 2019 model, a smaller student of GPT-2 with 82 million parameters, 6 blocks and 12 attention heads per block. It is about 85 MB, and it runs right here in your browser. Nothing you type leaves your device.

Read the papers: GPT-1, Radford et al., 2018 · GPT-2, Radford et al., 2019 · GPT-3, Brown et al., 2020 · DistilBERT, the method behind DistilGPT-2, Sanh et al., 2019.

ChatGPT

A model trained only to guess the next word does exactly that. Ask GPT-3 a question and it might continue with more questions, because that is what text on the internet often does next. ChatGPT added two more rounds of training on top of the same next-word machine.

Instruction tuning: people wrote examples of good answers to requests, and the model was trained on them. Learning from human feedback (RLHF): people compared pairs of the model's answers and picked the better one, a second model learned their taste, and the chatbot was trained to score well with it. Then OpenAI wrapped it in the chat window you know. It reached a million users in five days.

Your turn: watch it think

The chat is DistilGPT-2 now, the real model, running on your device. It never got ChatGPT's extra training, so it doesn't answer you. It continues your text. That is the raw next-word machine from 2019, exactly what sits underneath every chatbot.

Type the start of a sentence, like "I need help paying my". Then turn on the chat format below and ask it a question, the way people coaxed answers out of these models before ChatGPT.

These probabilities are the real ones, computed in your browser. Attention weights inside this model aren't shown, because the browser version of the model doesn't output them.