How GPT Actually Works: A Visual Guide to Transformers

I could use GPT models for months before I could actually picture one. I knew the words — tokens, embeddings, attention, softmax — but if you’d asked me to draw what happens between “type a prompt” and “get a word back,” I’d have waved my hands and changed the subject.

The thing that finally fixed that for me was 3Blue1Brown’s But what is a GPT?. So this is me writing the explanation I wish I’d had — one continuous story from raw text to a sampled next token, with every calculation step on its own line and the real GPT-3 numbers attached wherever the full model would be too big to write out.

What does a GPT actually do?

GPT = Generative Pre-trained Transformer. Generative: it writes text. Pre-trained: it learned from a huge pile of data first. Transformer: the architecture from the 2017 paper Attention Is All You Need.

But the loop underneath all of that is embarrassingly simple. Given some text, the model outputs a probability distribution over the next token. Pick one token from that distribution, glue it on, and run again:

flowchart LR
    A[Input text] --> B[Model predicts<br/>distribution over<br/>next token]
    B --> C[Sample one token<br/>from the distribution]
    C --> D[Append token<br/>to the text]
    D --> A

That predict → sample → append loop is what you watch when ChatGPT types one word at a time. Everything else in this post is just the math inside the “predict” box — and I’m going to walk it in order: tokens → embeddings → dot products → softmax → temperature, then tie it together with one complete numeric example.

How does text become tokens?

Before any number touches the transformer, the string gets chopped into tokens — sometimes whole words, often sub-word pieces. Take the sentence from the video:

To date, the cleverest thinker of all time was …

A tokenizer might split it like this:

**To date , the cle ve rest thinker of all time was …**

“cleverest” becomes cle | ve | rest. The model never sees letters — only integer token IDs. In GPT-3 the vocabulary has exactly:

$$ V = 50{,}257 \text{ possible tokens} $$

I’ll use clean word-sized tokens in the worked example below so the arithmetic stays readable, but the real unit is always the token ID.

How do token IDs become vectors?

Each token ID gets mapped to a vector — a long list of numbers — by looking up a column in the embedding matrix $W_E$.

In GPT-3:

$$ \text{embedding dimension } d = 12{,}288 $$

$$ W_E \in \mathbb{R}^{12{,}288 \times 50{,}257} $$

Each column is one token’s vector. “Embedding token $t$” means: grab column $t$ of $W_E$. The values start random and get learned during training. That one matrix alone holds:

$$ 12{,}288 \times 50{,}257 = 617{,}558{,}016 \text{ weights} $$

617 million parameters before a single attention layer — and the full model has about 175 billion. I kept a running tally while watching the video, and that’s what made the scale click for me.

What does training do to those vectors?

After training, directions in embedding space mean something. Nobody programmed this; gradient descent just found it useful.

Gender lives in a direction. $\text{woman} - \text{man}$ is roughly $\text{queen} - \text{king}$, so $\text{king} + (\text{woman} - \text{man})$ lands near queen.
Nationality generalizes. $\text{Italy} - \text{Germany} + \text{Hitler}$ lands near Mussolini.
Plurality is a direction. $\text{cats} - \text{cat}$ points along “more than one.”

A raw embedding is still dumb on its own — the lookup for mole is identical whether you mean the animal, the skin mark, or the chemistry unit. The transformer’s middle layers fix that by letting each vector soak up context from its neighbors. GPT-3 can hold 2,048 tokens at once; after 96 attention + MLP blocks, the last token’s vector has absorbed information from everything before it. I’ll save how that mixing works for the attention deep dive — but the tool it uses everywhere is the dot product.

Why are dot products the similarity score?

To ask “how aligned are two vectors?” you take the dot product:

$$ \vec{a} \cdot \vec{b} = \sum_i a_i b_i $$

Positive → they point the same way (related). Zero → perpendicular (unrelated). Negative → opposite.

Concrete 2-D example. Let:

$$ \vec{a} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}, \quad \vec{b} = \begin{bmatrix} 1 \\ 2 \end{bmatrix} $$

Multiply coordinate by coordinate:

$$ a_1 b_1 = 3 \times 1 = 3 $$

$$ a_2 b_2 = 4 \times 2 = 8 $$

Sum:

$$ \vec{a} \cdot \vec{b} = 3 + 8 = 11 $$

In GPT-3’s embedding space each vector has 12,288 coordinates, so the sum has 12,288 terms — same idea, just longer. Attention uses dot products to score “how much should token $j$ listen to token $i$?” — that’s the engine I unpack in how attention works in transformers. For predicting the next token, the model eventually takes the last context-rich vector and runs one more matrix multiply.

How does the model score every possible next token?

After all the blocks, take the final vector $\vec{h}$ for the last token — shape $12{,}288 \times 1$ in GPT-3. Multiply by the unembedding matrix $W_U$:

$$ W_U \in \mathbb{R}^{50{,}257 \times 12{,}288} $$

$$ \vec{z} = W_U \, \vec{h} $$

Each entry of $\vec{z}$ is a raw score — a logit — for one vocabulary token. You get 50,257 of them. $W_U$ is basically $W_E$ flipped, so it adds another:

$$ 50{,}257 \times 12{,}288 = 617{,}558{,}016 \text{ weights} $$

Just over 1 billion parameters on the two ends alone; the other ~174 billion sit in the attention and MLP blocks in the middle.

Why only the last vector at inference? During training the model predicts the next token at every position in one pass — thousands of learning signals. At inference you only care about what comes next, so you read off the last one.

What does softmax turn those scores into?

Logits can be any real number and they don’t sum to 1. Softmax converts them into a valid probability distribution. For logits $z_1, z_2, \ldots, z_V$:

$$ \text{softmax}(\vec{z})_i = \frac{e^{z_i}}{\sum_{j=1}^{V} e^{z_j}} $$

Every output lands in $(0, 1)$ and the outputs sum to 1. I’ll walk a tiny three-token case so every step is visible.

Suppose the vocabulary is [cat, dog, runs] and the logits are:

$$ \vec{z} = \begin{bmatrix} 2 \\ -1 \\ 1 \end{bmatrix} $$

Step 1 — exponentiate each logit.

$$ e^{z_{\text{cat}}} = e^{2} = 7.389056 $$

$$ e^{z_{\text{dog}}} = e^{-1} = 0.367879 $$

$$ e^{z_{\text{runs}}} = e^{1} = 2.718282 $$

Step 2 — sum for the normalizer.

$$ Z = 7.389056 + 0.367879 + 2.718282 $$

$$ Z = 10.475217 $$

Step 3 — divide each exponential by $Z$.

$$ P(\text{cat}) = \frac{7.389056}{10.475217} = 0.7054 $$

$$ P(\text{dog}) = \frac{0.367879}{10.475217} = 0.0351 $$

$$ P(\text{runs}) = \frac{2.718282}{10.475217} = 0.2595 $$

Check the sum:

$$ 0.7054 + 0.0351 + 0.2595 = 1.0000 $$

Greedy decoding picks cat (highest probability). Sampling rolls a weighted die with those three probabilities. In the real model, the same softmax runs over all 50,257 tokens.

What does temperature change in those probabilities?

Temperature $T$ rescales logits before softmax:

$$ \text{softmax}(\vec{z}/T)_i = \frac{e^{z_i/T}}{\sum_j e^{z_j/T}} $$

Low $T$ sharpens the distribution toward the top token — safe, repetitive, near-deterministic. High $T$ flattens it — more surprising, more creative, occasionally unhinged. Same logits $\vec{z} = [2, -1, 1]$, now with $T = 0.5$.

Step 1 — divide each logit by $T$.

$$ \frac{z_{\text{cat}}}{T} = \frac{2}{0.5} = 4 $$

$$ \frac{z_{\text{dog}}}{T} = \frac{-1}{0.5} = -2 $$

$$ \frac{z_{\text{runs}}}{T} = \frac{1}{0.5} = 2 $$

Step 2 — exponentiate.

$$ e^{4} = 54.598150 $$

$$ e^{-2} = 0.135335 $$

$$ e^{2} = 7.389056 $$

Step 3 — normalizer.

$$ Z_{T=0.5} = 54.598150 + 0.135335 + 7.389056 = 62.122541 $$

Step 4 — probabilities.

$$ P(\text{cat}) = \frac{54.598150}{62.122541} = 0.8790 $$

$$ P(\text{dog}) = \frac{0.135335}{62.122541} = 0.0022 $$

$$ P(\text{runs}) = \frac{7.389056}{62.122541} = 0.1189 $$

Compare to $T = 1$: cat went from 70.5% to 87.9%. The API slider I’d been blindly nudging is just this — rescale logits, then softmax.

Can we walk through one complete next-token prediction?

Yes. I’ll trace the full pipeline on a toy prompt, naming the GPT-3 dimensions wherever the real model would use them.

Prompt: The cat

Step 1 — tokenize

$$ \text{tokens} = [\text{The}, \text{ cat}] $$

(GPT-3 would assign integer IDs; I’ll keep the words.)

Step 2 — embed each token

Toy embedding dimension $d = 2$ instead of 12,288. Suppose $W_E$ has learned:

$$ \vec{E}_{\text{The}} = \begin{bmatrix} 0.5 \\ 0.2 \end{bmatrix}, \quad \vec{E}_{\text{ cat}} = \begin{bmatrix} 1.0 \\ -0.5 \end{bmatrix} $$

In GPT-3 this lookup is a column grab from a $12{,}288 \times 50{,}257$ matrix — same operation, 12,288 numbers per token.

Step 3 — mix context (attention + MLP, ×96)

The middle of the model updates every vector so cat knows it follows The. After all blocks, the last vector might become:

$$ \vec{h} = \begin{bmatrix} 2.0 \\ -1.0 \end{bmatrix} $$

This is where attention does the heavy lifting — dot products between queries and keys, softmax over scores, weighted sum of values. I’m skipping the 96-layer detail here; the output is one context-soaked vector $\vec{h}$.

Step 4 — unembed to logits

Toy vocabulary: [cat, dog, runs]. Unembedding matrix:

$$ W_U = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{bmatrix} $$

Row 0 → cat, row 1 → dog, row 2 → runs. Each logit is a dot product of one row with $\vec{h}$:

$$ z_{\text{cat}} = W_U[0] \cdot \vec{h} = (1)(2) + (0)(-1) = 2 $$

$$ z_{\text{dog}} = W_U[1] \cdot \vec{h} = (0)(2) + (1)(-1) = -1 $$

$$ z_{\text{runs}} = W_U[2] \cdot \vec{h} = (1)(2) + (1)(-1) = 1 $$

$$ \vec{z} = \begin{bmatrix} 2 \\ -1 \\ 1 \end{bmatrix} $$

In GPT-3, $W_U$ has 50,257 rows and each logit is a 12,288-term dot product.

Step 5 — softmax at $T = 1$

$$ e^{2} = 7.389056 $$

$$ e^{-1} = 0.367879 $$

$$ e^{1} = 2.718282 $$

$$ Z = 7.389056 + 0.367879 + 2.718282 = 10.475217 $$

$$ P(\text{cat}) = 7.389056 / 10.475217 = 0.7054 $$

$$ P(\text{dog}) = 0.367879 / 10.475217 = 0.0351 $$

$$ P(\text{runs}) = 2.718282 / 10.475217 = 0.2595 $$

Step 6 — sample and append

Sample runs (26% chance), append it: The cat runs. Feed the longer string back in. Repeat.

Step 7 — what training does to wrong guesses

If the correct next token were runs but the model gave $P(\text{runs}) = 0.2595$, the cross-entropy loss is:

$$ L = -\log(0.2595) = 1.347 $$

The gradient on logits is $p - y$:

$$ \begin{bmatrix} 0.7054 \\ 0.0351 \\ 0.2595 \end{bmatrix} - \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix} = \begin{bmatrix} 0.7054 \\ 0.0351 \\ -0.7405 \end{bmatrix} $$

Push down cat and dog, push up runs. Backprop carries that through $W_U$, every transformer block, and $W_E$. GPT is not a database returning stored sentences — it’s a parameterized probability machine estimating $P(\text{next token} \mid \text{previous tokens})$ over and over.

What is the whole pipeline in one picture?

flowchart LR
    A["Prompt text"] --> B["Tokenize<br/>(50,257 vocab)"]
    B --> C["Embed via W_E<br/>(617M params, d=12,288)"]
    C --> D["Attention + MLP<br/>blocks ×96<br/>(dot products inside)"]
    D --> E["Take last vector"]
    E --> F["Unembed via W_U<br/>(617M params)"]
    F --> G["Logits → Softmax<br/>(÷ temperature)"]
    G --> H["Probability over<br/>50,257 tokens"]
    H --> I["Sample next token"]
    I --> A

Underneath all of it, one mental model kept the architecture from feeling like magic: parameters are weights, and nearly every step is a matrix-vector multiply with occasional nonlinearities (like softmax) wedged in so the stack doesn’t collapse into one boring linear map.

Where should I go next?

The middle of that diagram — the part where dot products decide which tokens talk to which — is attention. I wrote the full walkthrough with queries, keys, values, masking, and the real GPT-3 head counts here: how attention works in transformers →.

Watch: But what is a GPT? — 3Blue1Brown · full text version on 3blue1brown.com · part of the Neural Networks series.