---
layout: post
title: "GPT Math Explained: The Full Forward Pass Beyond Attention"
date: 2026-06-24 10:00:00 +0530
tags: ["GPT", "Transformers", "Attention", "Backpropagation", "AdamW", "Deep Learning", "Training"]
category: "ai"
description: "How does GPT math actually work end to end? Follow one training step from token IDs through Q/K/V, multi-head attention, MLP, logits, cross-entropy loss, backprop, and AdamW — past O = Attention(Q,K,V)."
keywords: "GPT math explained, transformer forward pass, when do W_Q W_K W_V update, attention output not final, GPT training step math, multi-head attention math, causal mask softmax, AdamW transformer training"
image: /assets/images/og-gpt-math-beyond-attention.png
order: 12
---

The biggest mistake in most transformer explanations is that they stop at:

$$
O = \text{Attention}(Q,K,V)
$$

and make it seem like that is the final answer.

It is not.

In reality, $O$ is only one tiny intermediate tensor inside GPT. I spent months treating attention as the whole story — then I tried to trace where $W_Q$, $W_K$, and $W_V$ actually get updated during training and hit a wall. The update doesn't happen *inside* attention. It happens at the end of a chain that most diagrams never draw.

Below I follow the **actual mathematical flow** exactly as GPT does it — one sentence, one token sequence, real shapes. If [how attention works](/blog/attention-in-transformers-explained/) gave you the intuition for queries, keys, and values, this is the rest of the pipeline: embeddings through logits, loss, backprop, and the [AdamW](/blog/adam-optimizer-explained/) step that finally moves those weight matrices.

```mermaid
flowchart TB
    TOK["Token IDs"] --> EMB["Embedding lookup"]
    EMB --> PE["+ Positional embedding"]
    PE --> QKV["Q = XW_Q, K = XW_K, V = XW_V"]
    QKV --> MHA["Multi-head attention + mask + softmax"]
    MHA --> PROJ["Output projection W_O"]
    PROJ --> RES1["Residual + LayerNorm"]
    RES1 --> MLP["MLP 768→3072→768"]
    MLP --> RES2["Residual + LayerNorm"]
    RES2 --> BLOCKS["× n_layer blocks"]
    BLOCKS --> LN["Final LayerNorm"]
    LN --> LOG["Logits = H E^T"]
    LOG --> CE["Cross-entropy loss"]
    CE --> BP["Backprop ∂L/∂W_Q, ∂L/∂W_K, ∂L/∂W_V"]
    BP --> ADAM["AdamW update"]
```

---

## Step 1: Training Data

Suppose the text is:

$$
\text{"The capital of France is Paris"}
$$

Tokenizer converts this into token ids:

$$
[1452,\,8912,\,302,\,7811,\,427,\,9204]
$$

GPT does not see words.

GPT only sees integers.

---

## Step 2: Input and Target

For next-token prediction:

Input:

$$
x = [1452,\,8912,\,302,\,7811,\,427]
$$

Target:

$$
y = [8912,\,302,\,7811,\,427,\,9204]
$$

Meaning:

$$
1452 \rightarrow 8912
$$

$$
8912 \rightarrow 302
$$

$$
302 \rightarrow 7811
$$

etc.

Every token tries to predict the next token.

---

## Step 3: Embedding Lookup

Suppose

$$
d_{\text{model}} = 768
$$

Each token id is converted into a vector.

For example:

$$
1452
$$

becomes

$$
e_1 =
\begin{bmatrix}
0.12 \\
-0.41 \\
0.83 \\
\vdots \\
0.09
\end{bmatrix}
$$

with 768 numbers.

Similarly:

$$
8912 \rightarrow e_2
$$

$$
302 \rightarrow e_3
$$

etc.

Stacking them:

$$
E =
\begin{bmatrix}
e_1 \\
e_2 \\
e_3 \\
e_4 \\
e_5
\end{bmatrix}
$$

Shape:

$$
(5,\,768)
$$

(See [token embeddings](/blog/token-embeddings-explained/) for the lookup table mechanics.)

---

## Step 4: Positional Embedding

Attention itself has no idea whether a word came first or last.

So GPT adds position vectors.

Position 0:

$$
p_0
$$

Position 1:

$$
p_1
$$

Position 2:

$$
p_2
$$

etc.

Final input becomes:

$$
X = E + P
$$

where

$$
P =
\begin{bmatrix}
p_0 \\
p_1 \\
p_2 \\
p_3 \\
p_4
\end{bmatrix}
$$

Now every token knows:

* what word it is
* where it is

([Positional encoding](/blog/positional-encoding-explained/) covers why we add rather than concatenate.)

---

## Step 5: Create Q, K, V

This is where attention starts.

GPT owns three learned matrices:

$$
W_Q
$$

$$
W_K
$$

$$
W_V
$$

Each has shape

$$
768 \times 768
$$

Initially they are random.

Not identity.

Not meaningful.

Just random numbers.

For example:

$$
W_Q(12,44) = 0.018
$$

$$
W_Q(50,91) = -0.021
$$

etc.

---

Then:

$$
Q = XW_Q
$$

$$
K = XW_K
$$

$$
V = XW_V
$$

Shapes:

$$
(5,\,768)
$$

for all three.

---

## What Are Q, K, V Actually?

Imagine token:

$$
\text{France}
$$

Its query vector asks:

$$
\text{"Which other words are useful for me?"}
$$

Its key vector says:

$$
\text{"What information do I offer?"}
$$

Its value vector contains:

$$
\text{"The information I actually carry."}
$$

---

## Step 6: Multi-Head Split

Suppose:

$$
d_{\text{model}} = 768
$$

and

$$
n_{\text{head}} = 12
$$

Then:

$$
d_{\text{head}} = \frac{768}{12} = 64
$$

GPT splits:

$$
Q,\,K,\,V
$$

into 12 pieces.

Each head gets:

$$
64
$$

dimensions.

---

Head 1 might learn:

$$
\text{grammar}
$$

Head 2:

$$
\text{subject-verb relationships}
$$

Head 3:

$$
\text{country-capital relationships}
$$

Head 4:

$$
\text{long-range dependencies}
$$

Nobody programs this.

Training discovers it automatically.

---

## Step 7: Compute Scores

Attention score matrix:

$$
S = QK^{\top}
$$

Shape:

$$
(5,\,5)
$$

Suppose the row for

$$
\text{Paris}
$$

becomes:

$$
[1.2,\,0.7,\,0.3,\,8.4,\,0.5]
$$

This means:

$$
\text{Paris}
$$

strongly matches

$$
\text{France}
$$

because

$$
8.4
$$

is much larger than everything else.

---

## Step 8: Scaling

Without scaling:

$$
QK^{\top}
$$

can become enormous.

For:

$$
d_{\text{head}} = 64
$$

we divide by:

$$
\sqrt{64} = 8
$$

Therefore:

$$
S = \frac{QK^{\top}}{8}
$$

This keeps softmax stable.

---

## Step 9: Causal Mask

GPT cannot see the future.

Suppose sequence:

$$
[\text{The},\,\text{capital},\,\text{of},\,\text{France},\,\text{is}]
$$

The token

$$
\text{France}
$$

cannot look at

$$
\text{Paris}
$$

because Paris has not been generated yet.

Mask:

$$
M =
\begin{bmatrix}
0 & -\infty & -\infty & -\infty & -\infty \\
0 & 0 & -\infty & -\infty & -\infty \\
0 & 0 & 0 & -\infty & -\infty \\
0 & 0 & 0 & 0 & -\infty \\
0 & 0 & 0 & 0 & 0
\end{bmatrix}
$$

Then:

$$
S = S + M
$$

Future positions become impossible.

---

## Step 10: Softmax

Suppose a row becomes:

$$
[1.2,\,0.4,\,8.4,\,0.3]
$$

Softmax converts it into probabilities:

$$
[0.0007,\,0.0003,\,0.9985,\,0.0005]
$$

Meaning:

$$
99.85\%
$$

attention goes to one token.

---

## Step 11: Attention Output

Attention matrix:

$$
A = \text{Softmax}(S)
$$

Output:

$$
O = AV
$$

This is the famous attention output.

For one token:

$$
o_i = a_{i1}v_1 + a_{i2}v_2 + \cdots + a_{in}v_n
$$

Weighted average of all value vectors.

---

This is where people stop.

But GPT is only beginning.

---

## Step 12: Output Projection

All heads are concatenated.

Suppose:

$$
12
$$

heads each produce

$$
64
$$

dimensions.

Concatenation gives:

$$
768
$$

dimensions again.

Then:

$$
OW_O
$$

mixes information from all heads.

---

## Step 13: Residual Connection

Input before attention:

$$
X
$$

Attention output:

$$
F(X)
$$

GPT computes:

$$
X_1 = X + F(X)
$$

The original information survives.

Attention only adds improvements.

---

## Step 14: LayerNorm

For every token vector:

$$
x = [x_1, x_2, \dots, x_{768}]
$$

Compute mean:

$$
\mu = \frac{1}{768} \sum_i x_i
$$

Compute variance:

$$
\sigma^2 = \frac{1}{768} \sum_i (x_i - \mu)^2
$$

Normalize:

$$
\hat{x} = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}}
$$

Then:

$$
y = \gamma \hat{x} + \beta
$$

where

$$
\gamma,\,\beta
$$

are learned.

---

## Step 15: MLP

Now comes the feedforward network.

Suppose:

$$
768 \rightarrow 3072 \rightarrow 768
$$

First:

$$
h = XW_1 + b_1
$$

Shape:

$$
(5,\,3072)
$$

Apply GELU:

$$
h = \text{GELU}(h)
$$

Then:

$$
Y = hW_2 + b_2
$$

Shape:

$$
(5,\,768)
$$

---

Attention answers:

$$
\text{"What information is important?"}
$$

MLP answers:

$$
\text{"What should I do with that information?"}
$$

---

## Step 16: Second Residual

Again:

$$
X_2 = X_1 + \text{MLP}(X_1)
$$

Now one transformer block is finished.

---

## Step 17: Repeat Many Times

Suppose:

$$
n_{\text{layer}} = 12
$$

Then:

$$
X \rightarrow \text{Block}_1 \rightarrow \text{Block}_2 \rightarrow \cdots \rightarrow \text{Block}_{12}
$$

Each layer gradually builds better representations.

Early layers:

$$
\text{words}
$$

Middle layers:

$$
\text{phrases}
$$

Later layers:

$$
\text{meaning}
$$

---

## Step 18: Final LayerNorm

After the last block:

$$
H = \text{LN}(X)
$$

Shape:

$$
(5,\,768)
$$

---

## Step 19: Vocabulary Projection

Now GPT converts hidden vectors into vocabulary scores.

Using tied embedding weights:

$$
\text{Logits} = H E^{\top}
$$

Shape:

$$
(5,\,V)
$$

where

$$
V = 50000
$$

for example.

For the final position:

$$
\begin{aligned}
\text{Paris} &= 12.4 \\
\text{London} &= 4.8 \\
\text{Berlin} &= 3.2 \\
\text{Dog} &= -1.7
\end{aligned}
$$

These are logits.

Not probabilities yet.

---

## Step 20: Softmax

$$
P = \text{Softmax}(\text{Logits})
$$

Maybe:

$$
\begin{aligned}
\text{Paris} &= 0.92 \\
\text{London} &= 0.05 \\
\text{Berlin} &= 0.02 \\
\text{Dog} &= 0.0001
\end{aligned}
$$

---

## Step 21: Loss

Correct answer:

$$
\text{Paris}
$$

Probability assigned:

$$
0.92
$$

Cross entropy:

$$
L = -\log(0.92)
$$

Small loss.

Good prediction.

---

If probability were:

$$
0.02
$$

then:

$$
L = -\log(0.02) \approx 3.91
$$

Large loss.

Bad prediction.

(See [cross-entropy loss](/blog/cross-entropy-loss-explained/) for why this is the right objective.)

---

## Step 22: The Part Everybody Misses

Now your question:

> When do $W_Q$, $W_K$, $W_V$ actually change?

Only now.

Not during attention.

Not during $O = AV$.

Not during softmax.

Only after loss exists.

---

Loss depends on:

$$
L
$$

which depends on

$$
\text{Logits}
$$

which depend on

$$
H
$$

which depends on

$$
O
$$

which depends on

$$
A
$$

which depends on

$$
QK^{\top}
$$

which depends on

$$
Q = XW_Q
$$

$$
K = XW_K
$$

$$
V = XW_V
$$

Therefore:

$$
L \rightarrow W_Q
$$

$$
L \rightarrow W_K
$$

$$
L \rightarrow W_V
$$

---

Backpropagation computes:

$$
\frac{\partial L}{\partial W_Q}
$$

$$
\frac{\partial L}{\partial W_K}
$$

$$
\frac{\partial L}{\partial W_V}
$$

These gradients answer:

> "If I slightly change this weight, how much will the loss change?"

---

## Step 23: AdamW Update

Suppose:

$$
W_Q(i,j) = 0.142
$$

Gradient:

$$
\frac{\partial L}{\partial W_Q(i,j)} = 0.8
$$

Learning rate:

$$
10^{-4}
$$

AdamW updates:

$$
W_Q(i,j) \leftarrow W_Q(i,j) - 10^{-4}(0.8)
$$

New value:

$$
0.14192
$$

Tiny change.

Learning happened.

---

## The Complete GPT Learning Equation

One training step is:

$$
\text{Tokens}
$$

$$
\downarrow
$$

$$
\text{Embeddings}
$$

$$
\downarrow
$$

$$
Q,\,K,\,V
$$

$$
\downarrow
$$

$$
\text{Multi-Head Attention}
$$

$$
\downarrow
$$

$$
\text{MLP}
$$

$$
\downarrow
$$

$$
12,\,24,\,48,\,\dots \text{ Transformer Blocks}
$$

$$
\downarrow
$$

$$
\text{Logits}
$$

$$
\downarrow
$$

$$
\text{Cross Entropy Loss}
$$

$$
\downarrow
$$

$$
\text{Backpropagation}
$$

$$
\downarrow
$$

$$
\frac{\partial L}{\partial W_Q},\,\frac{\partial L}{\partial W_K},\,\frac{\partial L}{\partial W_V}
$$

$$
\downarrow
$$

$$
\text{AdamW Update}
$$

$$
\downarrow
$$

$$
\text{New } W_Q,\,W_K,\,W_V
$$

The next forward pass uses these new matrices, producing different $Q$, $K$, $V$, different attention scores, different outputs, and hopefully a lower loss. That cycle repeats billions of times during GPT training.

I keep this diagram open when I'm debugging a training run — if loss isn't moving, the bug is almost never inside the softmax row. It's somewhere earlier in the chain, or the gradients never made it back to the weight that needed to move.