This page starts with the math that AI is built on, then follows one sentence all the way through a GPT, from the moment you type it to the moment an answer comes back. Every step is something you can see or try.
Topics link to the original research papers. Those papers are written for specialists, so expect math and jargon. This page is the plain-language version.
Every AI on this page is built from math that is taught in schools and universities. Here is the path, one class at a time.
Using letters to stand for unknown numbers, and finding them. If y = 2x + 1 and y = 7, then x = 3.
The math of angles, triangles and repeating waves, like sound waves and the seasons.
The math of change. It tells you how fast something is changing at any moment, called the derivative, and which direction to move to make a number go up or down.
What many students take after calculus. It is the math of lists and tables of numbers. A list of numbers is called a vector. A table of numbers is called a matrix. One multiplication can change every number in a huge table at once, and computer chips called GPUs are built to do exactly this very fast.
The math of uncertainty and data. It measures how likely something is and finds patterns across many examples.
Machine learning is a way to build a math formula from examples instead of writing the formula by hand. It uses each class above for a specific job:
Each dot is one apartment: its size and its monthly rent. The model is the formula rent = w × size + b. It starts with bad guesses for the two parameters w and b. Each training step uses calculus to nudge them so the line fits the dots better.
Example data, made up for teaching. The math is real: this is gradient descent, the same method used to train GPTs.
Read the papers
The model you just trained has 2 parameters. A GPT works on the same principle, with billions of parameters and trillions of words as examples. The rest of this page follows what happens inside one.
ChatGPT is a chat app built on a GPT. The three letters describe exactly what it is.
Read the papers
This is the real order of events every time you send a message.
Everything starts as plain text. Type your own sentence here and follow it down the page.
A model cannot read letters. A tokenizer cuts text into pieces called tokens. There is more than one way to cut. Here are the three main ones, all running live on your sentence.
Every letter, space and comma is its own token. Tiny vocabulary, nothing is ever unknown. But sentences become very long, and long means slow and expensive.
Each word is one token. Short and easy to read. But the vocabulary must hold every word ever, and any new word, name or typo becomes ? unknown.
The method GPT uses. Common words stay whole and rare words are built from familiar pieces, so nothing is unknown and sentences stay short. This one is real: it learned merges from a few paragraphs of text when this page loaded.
A dot (·) is a space that belongs to the token after it. Try a made-up word like “flurbish” and watch the word method fail.
t + h.th, and add it to the vocabulary.th + e becomes the. Thousands of rounds later, common words are single pieces.First merges this page learned:
Read the papers
| Tokenizer | Used by | Method | Vocabulary |
|---|---|---|---|
| GPT-2 tokenizer | GPT-2, GPT-3 | byte-level BPE | 50,257 |
| cl100k_base | GPT-3.5, GPT-4 | byte-level BPE | ~100,000 |
| o200k_base | GPT-4o | byte-level BPE | ~200,000 |
| Llama 3 tokenizer | Meta Llama 3 | BPE | 128,256 |
| WordPiece | Google BERT | WordPiece, similar to BPE | 30,522 |
| SentencePiece | Google T5 | SentencePiece | 32,000 |
Bigger vocabularies mean fewer tokens per sentence, especially for languages other than English, but a bigger model to store them. Anthropic's Claude uses its own tokenizer, and its details are not public.
The tokenizer looks up every piece in its vocabulary list and swaps it for that piece's position number, called a token ID. From here on, the model only ever sees numbers.
IDs from this page's small BPE vocabulary. A real GPT's IDs go up to about 100,000 or more.
Every token ID is swapped for a long list of numbers called an embedding. Think of it as an address on a giant map where words with similar meanings live near each other.
An illustration. Real embeddings have hundreds or thousands of numbers each, squeezed here onto a flat page. Notice that “tuition” sits between school and money, because it belongs to both.
Read the paper
Now the transformer reads all the tokens together. For each one, it asks which other words in the sentence matter for understanding it. This is called attention, and it is the T in GPT.
The scholarship was hard to find because it was buried at the bottom.
When the model reads “it,” how much attention goes to each earlier word?
Example numbers. A real model does this for every word, many times in parallel, across dozens of layers.
Read the paper
At the end, the model gives every token in its vocabulary a chance of coming next. Then it picks one. A setting called temperature controls how adventurous that pick is.
I need help paying my ___
Example scores. Low temperature almost always picks the top word. High temperature spreads the chances out, so unusual words get picked more often. This math is real: it is the same formula models use.
The picked token is added to the end of the text, and the whole trip from Step 2 runs again for the next one. A full answer is this loop, run hundreds or thousands of times.
Knowledge is
An example run with made-up choices. The model never plans the whole sentence. It only ever picks the next piece.
The model reads a huge amount of text and practices one game: guess the next token.
It guesses, checks the real next token, and adjusts its parameters, billions of tiny dials inside it, so the right answer gets a little more likely. After trillions of guesses, it has absorbed grammar, facts, styles and patterns. It also absorbs the gaps and biases in whatever it read.
Read the paper
Extra training on a smaller, focused set of examples.
A pre-trained model can continue any text, but it is not yet a helpful assistant. Fine-tuning on examples of good conversations teaches it to answer questions instead of, say, continuing your question with more questions.
Learning by trying things and getting a score.
Nobody shows the model the right answer. It tries something, gets a number back for how well it did, and slowly shifts toward whatever earns more. A dog learning tricks for treats works the same way.
Read the book
When the model finds a way to get a high score without doing what people wanted.
In a well-known 2016 example, an AI trained on a boat racing game discovered it could spin in circles hitting the same bonus targets forever. It beat human players' scores and never finished the race. The score was the goal, so the score is what it chased.
Read the paper
Using RL to teach a chatbot what good answers look like.
In RLHF, reinforcement learning from human feedback, people compare two answers and pick the better one. A separate reward model learns their taste, and the chatbot is trained to score well with it. In RLAIF, an AI does the comparing instead of people. HARK Entry 01 at Differend Labs tests whether an AI judge rates things the way people would.
Read the papers
Making sure AI does what people actually need, not only what its score says.
Reward hacking is the small version of the problem. Read the alignment problem for the full explanation.
An AI that could learn and do almost any thinking task a person can, across every subject, instead of being good at only some.
There is no agreed definition. Companies and researchers each describe it differently, and people strongly disagree about whether it will happen and when. Today's models are very strong at some tasks and still fail at others that people find easy. When someone says AGI, it is fair to ask which definition they mean.
Read the paper
An AI that takes actions on its own, step by step, to reach a goal.
A chatbot answers. An agent might search the web, fill in forms, send emails, or write and run code, with less checking by a person between steps. More actions mean more places for its goal and your needs to drift apart.
A way to prove something is true without revealing anything else.
Picture proving you found Waldo without showing where he is on the page. You cover the whole page with a huge sheet that has one tiny hole, and slide it so only Waldo shows through. Your friend sees Waldo, so they know you found him, but they learn nothing about where he is.
Real zero-knowledge proofs do this with math. They can prove you are over 18 without revealing your birthday. In AI, researchers are working on using them to prove a model ran correctly, or passed a safety test, without revealing private data or the model's secret parameters.
Read the paper
Research that opens up a trained model and works out what its parts do, and why.
In Step 4 and Step 5 above, each token became a list of numbers, and the transformer mixed those lists together, layer after layer. Each number computed along the way is called a neuron. A neuron takes the numbers from the layer before it, multiplies each by a weight, adds them up, and passes the total through a simple function. Every piece of text you give the model sets every neuron to some value. When the value is high, people say the neuron “fires.”
One way to study a neuron is to collect the texts or images that make it fire the most and look for what they share. Sometimes they all share one idea, such as pictures of cars. The idea a model has learned to detect is called a feature. A neuron that fires for one idea is called monosemantic, which means “one meaning.” A neuron that fires for several unrelated ideas is called polysemantic, “many meanings.” A well-known example from a 2020 study of an image model was a neuron that fired for both cat faces and the fronts of cars.
One explanation for polysemantic neurons is superposition. The idea is that a model needs to track more features than it has neurons, so it stores each feature as a direction spread across several neurons, and each neuron ends up carrying pieces of several unrelated features. Try it below.
This is a made-up example with real math. Three features share two neurons. Each feature is an arrow of length 1, and the two neurons read off how far the arrow points right (neuron 1) and up (neuron 2). Notice that neuron 2 gives the same reading, −0.50, for B and C, so neuron 2 alone cannot tell them apart. The dictionary line multiplies the two neuron readings by each arrow and adds them (a dot product). The matching arrow always scores 1.00 and the others score −0.50, so the three arrows recover all three features from only two neurons.
A sparse autoencoder (SAE) is a second model trained to do that recovery on a real model. It learns a long list of directions, called a dictionary, often with many more entries than the model has neurons, and it rewrites the model's activity using only a few entries at a time. “Sparse” means only a few entries are used at once. In 2024, Anthropic used this method to find millions of features inside its Claude 3 Sonnet model, including one that fired for the Golden Gate Bridge.
This matters for alignment. If people could read what a model is computing, they could check whether it is doing what people need, instead of judging it only by its answers. For large models, most of that inner computation is not yet understood.
Read the papers
How to tell what a piece of research has shown from what it hopes.
In 2026, Nick Cammarata of Goodfire, a company that does interpretability research, published a long essay called “Neural Networks Are Already Clean.” It argues that superposition was the wrong explanation for the messy neurons in early studies. In the essay's account, those neurons came from ordinary problems that can be fixed, such as weaknesses in how the models were built and in the tools used to inspect them. It also describes a pattern it calls recursive generalization: a neuron that learns one idea gradually takes on closely related ideas at its edges, so its extra jobs are connected rather than random. The essay reports that in early checks of Qwen3-32B, an open-weights language model, about 90% of the neuron behaviors audited had one main job. If this holds up, researchers could study a model's own neurons directly, instead of first training a second model to untangle them.
The essay drew wide attention and some pushback. Critics pointed out that it moves from measured results to much larger claims, such as this sentence from its summary:
“Highly speculative: this might be most of what we need, and interpretability might be far more tractable, possibly in the near term, than we’ve assumed.”
Three questions help with reading any claim like this.
Where the science stands: the essay calls its numbers preliminary and says a technical companion paper will give fuller details. Other researchers have found real limits in today's tools. In 2025, Google DeepMind's interpretability team reported that sparse autoencoders did worse than simpler methods on some practical tasks. Another 2025 study found that common automatic scores for how understandable a model's features are gave similar results for trained models and for models with random, untrained weights. So the fair summary is an interesting result, a large open question, and nothing settled yet.
Read the sources
Every other word, grouped by topic. Search finds words anywhere on this page.
pip install name does this. uv is a faster tool for the same job.No word matches that search. Email hello@differend.ai and it will be added.
Missing a word you keep hearing? Email hello@differend.ai and it will be added here.