--- layout: post title: "How GPT Actually Works: A Visual Guide to Transformers" date: 2026-06-04 10:00:00 +0530 tags: ["Transformers", "GPT", "LLM", "Deep Learning", "Word Embeddings", "Softmax", "3Blue1Brown"] category: "ai" description: "How does a GPT actually work? A plain-English, visual walkthrough of transformers — tokens, word embeddings, why directions in vector space mean something, dot products, softmax and temperature — with the real GPT-3 numbers worked out." keywords: "how does GPT work, how do transformers work, what is a GPT, how does chatgpt work, word embeddings explained, softmax temperature explained, GPT-3 parameters" image: /assets/images/og-what-is-gpt.png order: 9 --- I could *use* GPT models for months before I could actually *picture* one. I knew the words — tokens, embeddings, attention, softmax — but if you'd asked me to draw what happens between "type a prompt" and "get a word back," I'd have waved my hands and changed the subject. The thing that finally fixed that for me was [3Blue1Brown's *But what is a GPT?*](https://www.youtube.com/watch?v=wjZofJX0v4M). So this is me writing the explanation I wish I'd had — one continuous story from raw text to a sampled next token, with every calculation step on its own line and the real GPT-3 numbers attached wherever the full model would be too big to write out. {% include youtube.html id="wjZofJX0v4M" title="But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning by 3Blue1Brown" %} ## What does a GPT actually do? **GPT = Generative Pre-trained Transformer.** Generative: it writes text. Pre-trained: it learned from a huge pile of data first. Transformer: the architecture from the 2017 paper *Attention Is All You Need*. But the loop underneath all of that is embarrassingly simple. Given some text, the model outputs a **probability distribution over the next token**. Pick one token from that distribution, glue it on, and run again: ```mermaid flowchart LR A[Input text] --> B[Model predicts
distribution over
next token] B --> C[Sample one token
from the distribution] C --> D[Append token
to the text] D --> A ``` That **predict → sample → append** loop is what you watch when ChatGPT types one word at a time. Everything else in this post is just the math inside the "predict" box — and I'm going to walk it in order: **tokens → embeddings → dot products → softmax → temperature**, then tie it together with one complete numeric example. ## How does text become tokens? Before any number touches the transformer, the string gets chopped into **tokens** — sometimes whole words, often sub-word pieces. Take the sentence from the video: > **To date, the cleverest thinker of all time was ...** A tokenizer might split it like this: > **To | date | , | the | cle | ve | rest | thinker | of | all | time | was ...** "cleverest" becomes `cle | ve | rest`. The model never sees letters — only integer **token IDs**. In GPT-3 the vocabulary has exactly: $$ V = 50{,}257 \text{ possible tokens} $$ I'll use clean word-sized tokens in the worked example below so the arithmetic stays readable, but the real unit is always the token ID. ## How do token IDs become vectors? Each token ID gets mapped to a **vector** — a long list of numbers — by looking up a column in the **embedding matrix** $W_E$. In GPT-3: $$ \text{embedding dimension } d = 12{,}288 $$ $$ W_E \in \mathbb{R}^{12{,}288 \times 50{,}257} $$ Each column is one token's vector. "Embedding token $t$" means: grab column $t$ of $W_E$. The values start random and get learned during training. That one matrix alone holds: $$ 12{,}288 \times 50{,}257 = 617{,}558{,}016 \text{ weights} $$ 617 million parameters before a single attention layer — and the full model has about 175 billion. I kept a running tally while watching the video, and that's what made the scale click for me. ### What does training do to those vectors? After training, **directions in embedding space mean something.** Nobody programmed this; gradient descent just found it useful. - **Gender lives in a direction.** $\text{woman} - \text{man}$ is roughly $\text{queen} - \text{king}$, so $\text{king} + (\text{woman} - \text{man})$ lands near *queen*. - **Nationality generalizes.** $\text{Italy} - \text{Germany} + \text{Hitler}$ lands near *Mussolini*. - **Plurality is a direction.** $\text{cats} - \text{cat}$ points along "more than one." A raw embedding is still dumb on its own — the lookup for *mole* is identical whether you mean the animal, the skin mark, or the chemistry unit. The transformer's middle layers fix that by letting each vector **soak up context** from its neighbors. GPT-3 can hold **2,048 tokens** at once; after 96 attention + MLP blocks, the last token's vector has absorbed information from everything before it. I'll save *how* that mixing works for the [attention deep dive](/blog/attention-in-transformers-explained/) — but the tool it uses everywhere is the dot product. ## Why are dot products the similarity score? To ask "how aligned are two vectors?" you take the **dot product**: $$ \vec{a} \cdot \vec{b} = \sum_i a_i b_i $$ Positive → they point the same way (related). Zero → perpendicular (unrelated). Negative → opposite. Concrete 2-D example. Let: $$ \vec{a} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}, \quad \vec{b} = \begin{bmatrix} 1 \\ 2 \end{bmatrix} $$ Multiply coordinate by coordinate: $$ a_1 b_1 = 3 \times 1 = 3 $$ $$ a_2 b_2 = 4 \times 2 = 8 $$ Sum: $$ \vec{a} \cdot \vec{b} = 3 + 8 = 11 $$ In GPT-3's embedding space each vector has 12,288 coordinates, so the sum has 12,288 terms — same idea, just longer. Attention uses dot products to score "how much should token $j$ listen to token $i$?" — that's the engine I unpack in [how attention works in transformers](/blog/attention-in-transformers-explained/). For predicting the next token, the model eventually takes the **last** context-rich vector and runs one more matrix multiply. ## How does the model score every possible next token? After all the blocks, take the final vector $\vec{h}$ for the last token — shape $12{,}288 \times 1$ in GPT-3. Multiply by the **unembedding matrix** $W_U$: $$ W_U \in \mathbb{R}^{50{,}257 \times 12{,}288} $$ $$ \vec{z} = W_U \, \vec{h} $$ Each entry of $\vec{z}$ is a raw score — a **logit** — for one vocabulary token. You get 50,257 of them. $W_U$ is basically $W_E$ flipped, so it adds another: $$ 50{,}257 \times 12{,}288 = 617{,}558{,}016 \text{ weights} $$ Just over **1 billion** parameters on the two ends alone; the other ~174 billion sit in the attention and MLP blocks in the middle. Why only the last vector at inference? During training the model predicts the next token at *every* position in one pass — thousands of learning signals. At inference you only care about what comes next, so you read off the last one. ## What does softmax turn those scores into? Logits can be any real number and they don't sum to 1. **Softmax** converts them into a valid probability distribution. For logits $z_1, z_2, \ldots, z_V$: $$ \text{softmax}(\vec{z})_i = \frac{e^{z_i}}{\sum_{j=1}^{V} e^{z_j}} $$ Every output lands in $(0, 1)$ and the outputs sum to 1. I'll walk a tiny three-token case so every step is visible. Suppose the vocabulary is `[cat, dog, runs]` and the logits are: $$ \vec{z} = \begin{bmatrix} 2 \\ -1 \\ 1 \end{bmatrix} $$ **Step 1 — exponentiate each logit.** $$ e^{z_{\text{cat}}} = e^{2} = 7.389056 $$ $$ e^{z_{\text{dog}}} = e^{-1} = 0.367879 $$ $$ e^{z_{\text{runs}}} = e^{1} = 2.718282 $$ **Step 2 — sum for the normalizer.** $$ Z = 7.389056 + 0.367879 + 2.718282 $$ $$ Z = 10.475217 $$ **Step 3 — divide each exponential by $Z$.** $$ P(\text{cat}) = \frac{7.389056}{10.475217} = 0.7054 $$ $$ P(\text{dog}) = \frac{0.367879}{10.475217} = 0.0351 $$ $$ P(\text{runs}) = \frac{2.718282}{10.475217} = 0.2595 $$ Check the sum: $$ 0.7054 + 0.0351 + 0.2595 = 1.0000 $$ **Greedy decoding** picks `cat` (highest probability). **Sampling** rolls a weighted die with those three probabilities. In the real model, the same softmax runs over all 50,257 tokens. ## What does temperature change in those probabilities? **Temperature** $T$ rescales logits before softmax: $$ \text{softmax}(\vec{z}/T)_i = \frac{e^{z_i/T}}{\sum_j e^{z_j/T}} $$ Low $T$ sharpens the distribution toward the top token — safe, repetitive, near-deterministic. High $T$ flattens it — more surprising, more creative, occasionally unhinged. Same logits $\vec{z} = [2, -1, 1]$, now with $T = 0.5$. **Step 1 — divide each logit by $T$.** $$ \frac{z_{\text{cat}}}{T} = \frac{2}{0.5} = 4 $$ $$ \frac{z_{\text{dog}}}{T} = \frac{-1}{0.5} = -2 $$ $$ \frac{z_{\text{runs}}}{T} = \frac{1}{0.5} = 2 $$ **Step 2 — exponentiate.** $$ e^{4} = 54.598150 $$ $$ e^{-2} = 0.135335 $$ $$ e^{2} = 7.389056 $$ **Step 3 — normalizer.** $$ Z_{T=0.5} = 54.598150 + 0.135335 + 7.389056 = 62.122541 $$ **Step 4 — probabilities.** $$ P(\text{cat}) = \frac{54.598150}{62.122541} = 0.8790 $$ $$ P(\text{dog}) = \frac{0.135335}{62.122541} = 0.0022 $$ $$ P(\text{runs}) = \frac{7.389056}{62.122541} = 0.1189 $$ Compare to $T = 1$: `cat` went from 70.5% to 87.9%. The API slider I'd been blindly nudging is just this — rescale logits, then softmax. ## Can we walk through one complete next-token prediction? Yes. I'll trace the full pipeline on a toy prompt, naming the GPT-3 dimensions wherever the real model would use them. **Prompt:** `The cat` ### Step 1 — tokenize $$ \text{tokens} = [\text{The}, \text{ cat}] $$ (GPT-3 would assign integer IDs; I'll keep the words.) ### Step 2 — embed each token Toy embedding dimension $d = 2$ instead of 12,288. Suppose $W_E$ has learned: $$ \vec{E}_{\text{The}} = \begin{bmatrix} 0.5 \\ 0.2 \end{bmatrix}, \quad \vec{E}_{\text{ cat}} = \begin{bmatrix} 1.0 \\ -0.5 \end{bmatrix} $$ In GPT-3 this lookup is a column grab from a $12{,}288 \times 50{,}257$ matrix — same operation, 12,288 numbers per token. ### Step 3 — mix context (attention + MLP, ×96) The middle of the model updates every vector so `cat` knows it follows `The`. After all blocks, the **last** vector might become: $$ \vec{h} = \begin{bmatrix} 2.0 \\ -1.0 \end{bmatrix} $$ This is where [attention](/blog/attention-in-transformers-explained/) does the heavy lifting — dot products between queries and keys, softmax over scores, weighted sum of values. I'm skipping the 96-layer detail here; the output is one context-soaked vector $\vec{h}$. ### Step 4 — unembed to logits Toy vocabulary: `[cat, dog, runs]`. Unembedding matrix: $$ W_U = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{bmatrix} $$ Row 0 → `cat`, row 1 → `dog`, row 2 → `runs`. Each logit is a dot product of one row with $\vec{h}$: $$ z_{\text{cat}} = W_U[0] \cdot \vec{h} = (1)(2) + (0)(-1) = 2 $$ $$ z_{\text{dog}} = W_U[1] \cdot \vec{h} = (0)(2) + (1)(-1) = -1 $$ $$ z_{\text{runs}} = W_U[2] \cdot \vec{h} = (1)(2) + (1)(-1) = 1 $$ $$ \vec{z} = \begin{bmatrix} 2 \\ -1 \\ 1 \end{bmatrix} $$ In GPT-3, $W_U$ has 50,257 rows and each logit is a 12,288-term dot product. ### Step 5 — softmax at $T = 1$ $$ e^{2} = 7.389056 $$ $$ e^{-1} = 0.367879 $$ $$ e^{1} = 2.718282 $$ $$ Z = 7.389056 + 0.367879 + 2.718282 = 10.475217 $$ $$ P(\text{cat}) = 7.389056 / 10.475217 = 0.7054 $$ $$ P(\text{dog}) = 0.367879 / 10.475217 = 0.0351 $$ $$ P(\text{runs}) = 2.718282 / 10.475217 = 0.2595 $$ ### Step 6 — sample and append Sample `runs` (26% chance), append it: `The cat runs`. Feed the longer string back in. Repeat. ### Step 7 — what training does to wrong guesses If the correct next token were `runs` but the model gave $P(\text{runs}) = 0.2595$, the cross-entropy loss is: $$ L = -\log(0.2595) = 1.347 $$ The gradient on logits is $p - y$: $$ \begin{bmatrix} 0.7054 \\ 0.0351 \\ 0.2595 \end{bmatrix} - \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix} = \begin{bmatrix} 0.7054 \\ 0.0351 \\ -0.7405 \end{bmatrix} $$ Push down `cat` and `dog`, push up `runs`. Backprop carries that through $W_U$, every transformer block, and $W_E$. GPT is not a database returning stored sentences — it's a parameterized probability machine estimating $P(\text{next token} \mid \text{previous tokens})$ over and over. ## What is the whole pipeline in one picture? ```mermaid flowchart LR A["Prompt text"] --> B["Tokenize
(50,257 vocab)"] B --> C["Embed via W_E
(617M params, d=12,288)"] C --> D["Attention + MLP
blocks ×96
(dot products inside)"] D --> E["Take last vector"] E --> F["Unembed via W_U
(617M params)"] F --> G["Logits → Softmax
(÷ temperature)"] G --> H["Probability over
50,257 tokens"] H --> I["Sample next token"] I --> A ``` Underneath all of it, one mental model kept the architecture from feeling like magic: **parameters are weights, and nearly every step is a matrix-vector multiply** with occasional nonlinearities (like softmax) wedged in so the stack doesn't collapse into one boring linear map. ## Where should I go next? The middle of that diagram — the part where dot products decide which tokens talk to which — is attention. I wrote the full walkthrough with queries, keys, values, masking, and the real GPT-3 head counts here: **[how attention works in transformers →](/blog/attention-in-transformers-explained/)**. **Watch:** [But what is a GPT? — 3Blue1Brown](https://www.youtube.com/watch?v=wjZofJX0v4M) · full text version on [3blue1brown.com](https://www.3blue1brown.com/lessons/gpt) · part of the [Neural Networks series](https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi).