The biggest mistake in most transformer explanations is that they stop at:
and make it seem like that is the final answer.
It is not.
In reality, $O$ is only one tiny intermediate tensor inside GPT. I spent months treating attention as the whole story — then I tried to trace where $W_Q$, $W_K$, and $W_V$ actually get updated during training and hit a wall. The update doesn’t happen inside attention. It happens at the end of a chain that most diagrams never draw.
Below I follow the actual mathematical flow exactly as GPT does it — one sentence, one token sequence, real shapes. If how attention works gave you the intuition for queries, keys, and values, this is the rest of the pipeline: embeddings through logits, loss, backprop, and the AdamW step that finally moves those weight matrices.
flowchart TB
TOK["Token IDs"] --> EMB["Embedding lookup"]
EMB --> PE["+ Positional embedding"]
PE --> QKV["Q = XW_Q, K = XW_K, V = XW_V"]
QKV --> MHA["Multi-head attention + mask + softmax"]
MHA --> PROJ["Output projection W_O"]
PROJ --> RES1["Residual + LayerNorm"]
RES1 --> MLP["MLP 768→3072→768"]
MLP --> RES2["Residual + LayerNorm"]
RES2 --> BLOCKS["× n_layer blocks"]
BLOCKS --> LN["Final LayerNorm"]
LN --> LOG["Logits = H E^T"]
LOG --> CE["Cross-entropy loss"]
CE --> BP["Backprop ∂L/∂W_Q, ∂L/∂W_K, ∂L/∂W_V"]
BP --> ADAM["AdamW update"]
Step 1: Training Data
Suppose the text is:
Tokenizer converts this into token ids:
GPT does not see words.
GPT only sees integers.
Step 2: Input and Target
For next-token prediction:
Input:
Target:
Meaning:
etc.
Every token tries to predict the next token.
Step 3: Embedding Lookup
Suppose
Each token id is converted into a vector.
For example:
becomes
with 768 numbers.
Similarly:
etc.
Stacking them:
Shape:
(See token embeddings for the lookup table mechanics.)
Step 4: Positional Embedding
Attention itself has no idea whether a word came first or last.
So GPT adds position vectors.
Position 0:
Position 1:
Position 2:
etc.
Final input becomes:
where
Now every token knows:
- what word it is
- where it is
(Positional encoding covers why we add rather than concatenate.)
Step 5: Create Q, K, V
This is where attention starts.
GPT owns three learned matrices:
Each has shape
Initially they are random.
Not identity.
Not meaningful.
Just random numbers.
For example:
etc.
Then:
Shapes:
for all three.
What Are Q, K, V Actually?
Imagine token:
Its query vector asks:
Its key vector says:
Its value vector contains:
Step 6: Multi-Head Split
Suppose:
and
Then:
GPT splits:
into 12 pieces.
Each head gets:
dimensions.
Head 1 might learn:
Head 2:
Head 3:
Head 4:
Nobody programs this.
Training discovers it automatically.
Step 7: Compute Scores
Attention score matrix:
Shape:
Suppose the row for
becomes:
This means:
strongly matches
because
is much larger than everything else.
Step 8: Scaling
Without scaling:
can become enormous.
For:
we divide by:
Therefore:
This keeps softmax stable.
Step 9: Causal Mask
GPT cannot see the future.
Suppose sequence:
The token
cannot look at
because Paris has not been generated yet.
Mask:
Then:
Future positions become impossible.
Step 10: Softmax
Suppose a row becomes:
Softmax converts it into probabilities:
Meaning:
attention goes to one token.
Step 11: Attention Output
Attention matrix:
Output:
This is the famous attention output.
For one token:
Weighted average of all value vectors.
This is where people stop.
But GPT is only beginning.
Step 12: Output Projection
All heads are concatenated.
Suppose:
heads each produce
dimensions.
Concatenation gives:
dimensions again.
Then:
mixes information from all heads.
Step 13: Residual Connection
Input before attention:
Attention output:
GPT computes:
The original information survives.
Attention only adds improvements.
Step 14: LayerNorm
For every token vector:
Compute mean:
Compute variance:
Normalize:
Then:
where
are learned.
Step 15: MLP
Now comes the feedforward network.
Suppose:
First:
Shape:
Apply GELU:
Then:
Shape:
Attention answers:
MLP answers:
Step 16: Second Residual
Again:
Now one transformer block is finished.
Step 17: Repeat Many Times
Suppose:
Then:
Each layer gradually builds better representations.
Early layers:
Middle layers:
Later layers:
Step 18: Final LayerNorm
After the last block:
Shape:
Step 19: Vocabulary Projection
Now GPT converts hidden vectors into vocabulary scores.
Using tied embedding weights:
Shape:
where
for example.
For the final position:
These are logits.
Not probabilities yet.
Step 20: Softmax
Maybe:
Step 21: Loss
Correct answer:
Probability assigned:
Cross entropy:
Small loss.
Good prediction.
If probability were:
then:
Large loss.
Bad prediction.
(See cross-entropy loss for why this is the right objective.)
Step 22: The Part Everybody Misses
Now your question:
When do $W_Q$, $W_K$, $W_V$ actually change?
Only now.
Not during attention.
Not during $O = AV$.
Not during softmax.
Only after loss exists.
Loss depends on:
which depends on
which depend on
which depends on
which depends on
which depends on
which depends on
Therefore:
Backpropagation computes:
These gradients answer:
“If I slightly change this weight, how much will the loss change?”
Step 23: AdamW Update
Suppose:
Gradient:
Learning rate:
AdamW updates:
New value:
Tiny change.
Learning happened.
The Complete GPT Learning Equation
One training step is:
The next forward pass uses these new matrices, producing different $Q$, $K$, $V$, different attention scores, different outputs, and hopefully a lower loss. That cycle repeats billions of times during GPT training.
I keep this diagram open when I’m debugging a training run — if loss isn’t moving, the bug is almost never inside the softmax row. It’s somewhere earlier in the chain, or the gradients never made it back to the weight that needed to move.
Support the writing
If this post helped, a coffee keeps the deep dives, paper picks, and news digests coming.
Buy Me a Coffee