A step-by-step walkthrough
What actually happens
inside a transformer?
We'll follow one sentence — the same one you can't parse without context — through every stage of the pipeline: from raw text, to numbers, to a model that knows exactly which word "it" refers to.
Scroll down, or jump to any stage on the left. Most steps are interactive — Step 4 flips "big"/"small"; try tokenizers, embedding clusters, layer depth, and prediction examples too.
Step 01
Tokenization
The model can't read words — it reads tokens, small chunks of text. The first step splits input into a numbered sequence the rest of the pipeline can process.
Raw text is split into a sequence of token IDs. Each ID points to a row in an embedding table learned during training. There is no single universal tokenizer — models pick byte-pair encoding (BPE), word-piece, or other schemes.
One token per word (simplified demo)— 13 tokens for our sentence.
- GPT-2~50k vocab · BPE
- BERTword-piece splits
- LLaMA 232k vocab
- GPT-4 class100k+ vocab
13 tokens — whole words and punctuation kept together where possible.
Switch modes above to see how the same sentence splits differently. GPT-style models typically use BPE; character-level is rare at scale because sequences become very long.
Step 02
Embeddings
Each token ID becomes a vector — hundreds or thousands of numbers that encode meaning learned from training data.
Each token ID is mapped to a dense vector — a learned lookup table, not a hand-written definition. Similar words end up near each other in this space after training.
Vectors show 4 of 768+ numbers per token. Click a chip to highlight an illustrative similarity cluster.
- BERT-base768 dims
- GPT-2768 dims
- LLaMA-7B4,096 dims
- GPT-3 (175B)12,288 dims
"trophy" groups with "suitcase" - illustrative physical objects cluster in embedding space.
Real models use hundreds of dimensions; we project to 2D so you can see that geometry encodes meaning — physical objects cluster together after training, not because we labeled them.
Step 03
Positional encoding
Parallel processing has no built-in order. A positional signal is added to each embedding so the model knows where each token sits.
Attention reads all tokens in parallel, so position is not implicit. A positional pattern is added to each embedding — sinusoidal in the original Transformer, rotary (RoPE) in many modern models.
Click any token to inspect its position encoding. Same word, different slot → different vector added to the embedding.
- BERT512 tokens max
- GPT-32,048–4,096
- GPT-4128k+ tokens
Position 8 for "it": P(8) = [0.99, 1.00, 0.00, 1.00] added to its embedding before attention.
Toggle "it" positions to see why position must be injected — the embedding for "it" alone cannot tell the model where it sits in the sentence.
Step 04 — interactive
Self-attention
Every token asks every other token: "how relevant are you to me?" Flip the ending below and watch where "it" points.
Every token projects into query, key, and value vectors. Each query scores every key; those scores become weights that mix values. Step 05 runs this many times in parallel as separate heads.
Click a query token on the left to see which key tokens it attends to. Thicker arrows = higher weight.
With "big,"the thing that didn't fit is the trophy — so "it" attends strongly to trophy and barely to suitcase.
Step 05
Multi-head attention
One head captures one relationship type. Real models run many heads in parallel — often 8, 12, or more — then merge the results.
Step 04 showed one attention pattern. Multi-head attention runs that same operation 8 times in parallel — each head uses its own learned projections (different Q, K, V matrices) on a slice of the embedding, so each head can specialize.
This demo shows 8 illustrative heads for our sentence. Real models use many more — the count is a hyperparameter called num_heads, not fixed at 3:
- BERT-base12 heads
- GPT-212 heads
- LLaMA-7B32 heads
- GPT-3 (175B)96 heads
Head 1 · local grammar
Neighboring words — article + noun
Head 2 · coreference
Pronoun → antecedent (Step 04)
Head 3 · negation scope
Negation reaching the verb
Head 4 · preposition phrase
Preposition + object
Head 5 · article–noun
Second article pairing
Head 6 · clause bridge
Subordinator → main clause
Head 7 · modifier scope
Degree word + adjective
Head 8 · long-range syntax
Subject linked across the sentence
All 8head outputs are concatenated back into one vector per token — grammar, coreference, negation, and other patterns combined. We don't design these roles; they emerge during training.
Step 06
Feed-forward network
After attention mixes tokens together, each position passes through the same small MLP on its own — refining what it learned from context.
After multi-head attention mixes information between tokens, each token vector passes through the same two-layer MLP — independently. Attention gathers context; the FFN refines what each token now knows on its own.
Click a token, then step through the FFN. The same weights process every position.
- BERT-base768 → 3,072 → 768
- GPT-24× expansion typical
- LLaMA-7B4,096 → 11,008
"it" after attention — 8 heads mixed context into this vector.
Inside one transformer block
Every token gets this private step — attention mixes tokens together, then each one thinks alone through the FFN.
Step 07
Stacking layers
Attention + feed-forward form one layer. Production models stack this block dozens of times — depth is a hyperparameter, not a fixed small number.
One round of multi-head attention + feed-forward (with residual connections and layer norm) is a layer, or block. Real models repeat this block many times — each layer refines the representation using what earlier layers already computed.
Click a layer to see what it emphasizes. Full model depth: 3 blocks.
- BERT-base12 layers
- GPT-2 XL48 layers
- LLaMA-7B32 layers
- GPT-3 (175B)96 layers
Articles, neighbors, short syntax — “The” + “trophy”, “in” + “suitcase”.
Attention head count also grows with model size:
- BERT-base12 heads
- GPT-212 heads
- LLaMA-7B32 heads
- GPT-3 (175B)96 heads
Early layers: grammar and local structure. Later layers: abstract understanding built on top — click each layer above to compare.
Step 08
Predicting the next word
The final layer projects onto the full vocabulary — a probability for every possible next token.
After the final layer, the model projects each token's vector onto the entire vocabulary and applies softmax — a probability for every possible next token. Training teaches the model to assign high probability to the token that actually came next.
Switch examples — only the last token's distribution drives the next-word prediction.
- GPT-250,257 outputs
- LLaMA 232,000 outputs
- GPT-4 class100k+ outputs
"The cat sat on the___"
Classic filler example — model predicts a surface noun after “on the”.
Everything before this — tokenizing, embedding, positional encoding, 8 parallel attention heads per layer, feed-forward blocks, stacked dozens of times — exists to sharpen this one distribution.