All topics

LLMs

Every post tagged “LLMs”, newest first.

9 posts
LLMs 11 min

What We Know About How Jev Works, and How the Open Copies Build It

TypeSafe hasn't published Jev's architecture. Its docs and the open models that copy its interface still show a lot about how a model can return calibrated typed answers without writing text.

LLMs 7 min

Colibri Explained: How a 744B LLM Runs From Disk

Colibri runs GLM-5.2 on ordinary machines by keeping routed experts on disk. I read the source to see how routing, caching and pinning work, and where the claims need care.

LLMs 25 min

DeepSeek V4 Inside: One Token Through Every Block

How does DeepSeek V4 process one token? I follow the byte p in deepseek through CSA, HCA, mHC, MoE, and MTP on a toy width, with Flash-0731 sizes.

Transformers 16 min

Looped Transformers Explained: Recurrent Depth and Astra

What is a looped transformer? How recurrent depth reuses layers, why Nanbeige 4.2 runs a 22-layer stack twice, and why that does not hide chain of thought.

LLMs 13 min

Build a Mini LLM from Scratch in NumPy: RoPE, GQA, SwiGLU

How to build a mini LLM from scratch in NumPy: RoPE, GQA, QK-Norm, SwiGLU, tied embeddings, and a 3.87M chat companion with full architecture visuals.

LLMs 15 min

DeepSeek V4 Explained: Long-Context Engineering and Math

A verification-first look at DeepSeek V4 claims, plus the real math behind sparse attention, KV-cache scaling, and long-context training stability.

Attention 15 min

How Attention Works in Transformers: Queries, Keys, Values

How does attention work in transformers? Queries, keys, values, the attention pattern, masking, and multi-head attention, with real GPT-3 parameter counts.

Transformers 13 min

How GPT Actually Works: A Visual Guide to Transformers

How does a GPT actually work? A visual walkthrough of transformers: tokens, word embeddings, dot products, softmax and temperature, with real GPT-3 numbers.

LLMs 42 min

Build GPT-2 from Scratch in PyTorch: A Full Walkthrough

How to build GPT-2 from scratch in PyTorch: tokenization, causal self-attention, transformer blocks, weight tying, and a 124M training loop that runs.