I started this as notes on a third-party video about a supposed DeepSeek V4 release. That is not enough evidence for model specifications. As of June 11, 2026, I could not verify an official DeepSeek V4 paper, repository, model card, or announcement supporting the precise figures and benchmark claims below. They must therefore be treated as claims from the linked video, not established facts.
The useful part is the engineering underneath. The video stitches together several real ideas worth understanding on their own. I separate those ideas from the release story, work through the mathematics line by line, and show how I evaluate frontier-model claims before repeating them.
[!NOTE] New to how these models work under the hood? My write-up on how attention works sets up queries, keys, values, and the quadratic cost — the exact thing V4 spends most of its cleverness fighting. I’ll lean on it here.
How do I verify a DeepSeek V4 claim before citing it?
I run a short checklist before I repeat any frontier-model number:
- Find the developer’s official announcement and paper.
- Match the exact model name to a repository or model card.
- Check whether parameter counts mean total or active parameters.
- Separate maximum context support from measured retrieval quality.
- Read benchmark methodology, baselines, sampling settings, and contamination discussion.
- Treat third-party videos and screenshots as leads, not primary evidence.
Until those checks pass, the correct language is “the source claims,” not “the model uses” or “the paper proves.” The rest of this article follows that rule.
What does the source claim about DeepSeek V4?
Verification status: unconfirmed. The “1.6 trillion parameters,” “1 million tokens,” training-token count, architecture names, and benchmark results in this section are attributed to the third-party source. Do not cite them as official DeepSeek specifications without a primary source.
Two headline numbers, and both of them are the problem as much as the feature:
- 1.6 trillion parameters (the V4 Pro model). Parameters are the dials inside the model that store what it knows. More dials, more capacity — but also far harder to train without the whole thing falling over.
- 1 million tokens of context. That’s roughly 750,000 words — the entire Harry Potter series in the prompt, with the model still able to pull one specific number off one specific page. For agents, it means running for hours on a long task without losing the thread.
The catch is that neither of those is free. A 1.6T-parameter model wants to explode during training, and a 1M-token context wants to eat all your compute at inference. The source frames the whole design as a list of fixes for those two failure modes. I buy the engineering logic; I do not buy the release story without a primary source.
Why is a 1-million-token context so expensive?
Models don’t read like we do. For every new token, attention asks “how does this relate to every token before it?” — the mechanism from the 2017 Attention Is All You Need paper that everything since is built on.
For a 10-word sentence that’s ~10 comparisons. Nothing. But the cost grows with the square of the sequence length:
Plug in the numbers and the wall is obvious. For $n = 2{,}048$:
For $n = 100{,}000$:
For $n = 1{,}000{,}000$:
| Context length $n$ | Comparisons $n^2$ |
|---|---|
| 2,048 | ~4.2 million |
| 100,000 | ~10 billion |
| 1,000,000 | ~1 trillion |
Double the context, quadruple the work. At a million tokens even high-end hardware chokes. And compute isn’t the only victim — the model has to store the intermediate keys and values for everything it’s seen, the KV cache, and that grows linearly with context until it swallows your memory budget whole.
flowchart LR
A["New token"] --> B["Compare against<br/>every prior token"]
B --> C["O(n²) attention cost"]
B --> D["Store keys + values<br/>→ KV cache grows"]
C --> E["Compute wall"]
D --> E
So “just use a bigger context window” isn’t an engineering shrug. It’s the central bottleneck of the entire field.
How does sparse attention cut the pair count?
Dense causal attention over $n$ tokens forms roughly $n(n+1)/2$ usable query-key pairs — each query can only look at tokens at or before its position.
Start with $n = 1{,}000{,}000$:
That is about $5 \times 10^{11}$ pairs per head and layer in the dense case.
A sparse design lets each query attend to only a local window of width $w$ plus $r$ retrieved distant tokens. Pair count drops toward $n(w+r)$. Take the source’s illustrative values $w = 128$ and $r = 32$:
That is $1.6 \times 10^{8}$ pairs. Compare to the dense count:
Over 3,000 times fewer pairs on paper. This does not prove any claimed implementation achieves that speedup — routing, retrieval, memory movement, and kernel utilization add overhead. It shows why sparse patterns are attractive enough that I’d expect every long-context design to use something like them, whether or not V4 exists.
How much memory does a 1M-token KV cache actually need?
The KV cache stores one key vector and one value vector per token, per key/value head, per layer. For $L$ layers, $h_{kv}$ key/value heads, head dimension $d_h$, sequence length $n$, and $b$ bytes per stored value:
The leading factor $2$ is keys plus values. I’ll walk through a plausible toy config at $n = 1{,}000{,}000$ with FP16 ($b = 2$ bytes):
| Symbol | Value |
|---|---|
| $L$ (layers) | 32 |
| $h_{kv}$ (KV heads) | 8 |
| $d_h$ (head dim) | 128 |
| $n$ (sequence length) | 1,000,000 |
| $b$ (bytes per value) | 2 |
Step 1 — layers × heads:
Step 2 — multiply by head dimension:
Step 3 — multiply by sequence length (one vector per token):
Step 4 — keys + values:
Step 5 — bytes (FP16):
Step 6 — convert to GiB (divide by $1024$ three times):
One sequence. One user. No activations, no weights, no optimizer state — just the cache. Quantization, grouped-query attention, eviction, compression, or distributed storage is therefore essential at million-token scale. The math doesn’t care what the marketing slide says.
What is the claimed fix: three views of the same document?
The source says DeepSeek’s move isn’t to pick one clever attention trick — it’s to run three in parallel and let each cover a different weakness. Think of them as three simultaneous readings of the same text:
- Compressed long-range. The distant past gets aggressively squeezed — blocks of old tokens collapsed into compact summaries so the model can still reach far back without paying full price for every token.
- Selective retrieval. Compression alone loses precision. Ask about an exact phrase or a specific number buried half a million tokens ago and a one-block summary has thrown it away. So a second pathway selectively pulls back the specific distant tokens that matter, uncompressed.
- Sliding-window. The most recent stretch — say the last 128 tokens — is kept word-for-word, zero compression, full fidelity. Recent context is usually the most relevant, so it gets the most exact treatment.
flowchart TB
DOC["1M-token sequence"] --> C1["Compressed<br/>(distant past → summaries)"]
DOC --> C2["Selective retrieval<br/>(exact distant tokens)"]
DOC --> C3["Sliding window<br/>(last ~128 tokens, exact)"]
C1 --> MIX["Combine per layer"]
C2 --> MIX
C3 --> MIX
MIX --> OUT["Long-range recall +<br/>broad context +<br/>sharp local focus"]
By interleaving these layer by layer, the source claims V4 gets long-range retrieval, broad understanding, and sharp immediate focus at once — while sidestepping the quadratic blow-up. The payoff the video reports: against DeepSeek V3.2 (already one of the most compute-efficient models around), V4 Pro runs at 3.7× lower FLOPs — despite being bigger and having a far longer context. That’s the part I’d want to see in a reproducible benchmark before I repeat it.
How do you keep a trillion-parameter signal from exploding?
Depth is the other monster. Signals pass through hundreds of layers, and at every layer there’s a risk the values blow up or vanish. The classic fix is residual connections — bypass lanes that let the signal skip around a layer instead of being mangled by it. More recently the field moved to hyperconnections, which widen those lanes to give the signal more room.
The source’s finding: past a trillion parameters, even standard hyperconnections start throwing catastrophic spikes. The lanes aren’t enough. So they introduced manifold-constrained hyperconnections (MHC) — constrain the residual mixing onto a manifold of doubly stochastic matrices, where every row and every column sums to 1. Intuitively that means the layer can reroute and remix signal across lanes but can never amplify the total, so it can’t run away to infinity.
The problem is enforcing that constraint a trillion times. Their answer is the Sinkhorn–Knopp algorithm: before each layer, run a quick sequence of row-then-column normalizations — about 20 iterations — until the matrix satisfies the doubly-stochastic constraint.
flowchart LR
M["Raw mixing matrix"] --> R["Normalize rows"]
R --> C["Normalize columns"]
C --> Q{"Doubly stochastic?<br/>(~20 iters)"}
Q -- "no" --> R
Q -- "yes" --> L["Layer processes signal"]
Worked example: one Sinkhorn–Knopp step on a $2 \times 2$ matrix
Start with a positive matrix:
Row sums:
Row-normalize (divide each entry by its row sum):
Column sums of $A^{(1)}$:
Columns do not sum to 1 yet. Column-normalize (divide each column by its column sum):
Row sums are now off again. Alternating row and column normalization converges toward a matrix where both sums are one. Such a constraint can control mixing magnitude, but stability claims still require ablations, training curves, and reproducible code — none of which I’ve verified for V4.
At first glance bolting a 20-step loop onto every layer of a 1.6T-parameter model sounds insane — you just spent the whole design saving compute. The source claims fused GPU kernels, selective recomputation, and other deep GPU-level tricks shrink the entire overhead to 6.7% of runtime. Plausible in principle; unverified for this specific model.
How do you train a 33T-token run without the math blowing up?
Architecture is only half the story. The source says V4 Pro was trained on 33 trillion tokens, more text than a human could read in many lifetimes.
Dump all of that on a fresh model at once and it drowns, the way handing a beginner an advanced quantum mechanics textbook teaches them nothing. So the source describes a curriculum — start small, grow the context as the model stabilizes:
flowchart LR
S["4K tokens<br/>(grammar, syntax,<br/>local patterns)"] --> M1["16K"]
M1 --> M2["64K"]
M2 --> L["1M tokens<br/>(full long-range)"]
Early on the model only sees ~4,000-token windows — enough to learn local structure — then 16K, 64K, all the way to the full million. Curriculum learning at long context is a real technique; the specific schedule here is a claim.
At this scale the big threat is a loss spike: the math blows up and the run crashes. The usual industry fix is grim — stop, roll back to an earlier checkpoint, try again, burn the compute. The source says DeepSeek leaned instead on self-stabilizing training plus the Muon optimizer (the algorithm deciding how to nudge the dials when the model gets something wrong), squeezing more stable learning out of each step so the run doesn’t need babysitting.
What benchmarks does the source report?
Verification status: unconfirmed. Treat every bullet as “the video claims,” not “I measured.”
- Putnam 2025 — 120/120. A perfect score on one of the hardest undergraduate math competitions in the world.
- 1M-token retrieval beats Gemini 3.1 Pro. Pushed to the absolute limit of its context window, V4’s recall accuracy edges out Google’s latest, which also runs a million tokens.
- Outcompetes Opus 4.6 across the task suite the source reports.
- On the independent Artificial Analysis leaderboard it lands as the second-best open-source model, just under Kimi 2.6 and crowding the top closed models from Google, Anthropic, and OpenAI.
I’d want the eval scripts, the needle-in-a-haystack prompts, and the contamination discussion before I repeated any of this in a paper or a tweet.
Why is the engineering story worth understanding anyway?
What got me isn’t any single trick — it’s that there is no single trick. Interleaved sparse attention for the memory wall, manifold-constrained hyperconnections for the signal explosions, Sinkhorn–Knopp made cheap with fused kernels, a curriculum and a stable optimizer so a 33T-token run doesn’t die halfway. Dozens of small, sharp engineering decisions stacked until something impossible becomes routine. That’s the whole theme: not one breakthrough, a lot of disciplined ones.
And if the source is right about the open-weight release, they did it resource-constrained — smaller team, no top NVIDIA chips, a sliver of the compute the closed labs swing around — then put the weights on Hugging Face and wrote down the infrastructure details the closed labs treat as state secrets. For a field that’s been quietly closing up, a paper that just tells you how is the rarest thing in it. I haven’t confirmed that release happened; the engineering patterns are real either way.
Strong intuition means knowing what the equations imply and knowing what the available evidence does not establish. The sparse-attention math says million-token context is a memory and compute problem with known structure. The verification checklist says I still don’t know whether DeepSeek V4 solved it the way the video says.
Watch: The insane engineering of DeepSeek V4 — AI Search. If the attention mechanics felt fuzzy, start with how attention works in transformers.
Support the writing
If this post helped, a coffee keeps the deep dives, paper picks, and news digests coming.
Buy Me a Coffee