---
layout: post
title: "Colibri Explained: How a 744B LLM Runs From Disk"
date: 2026-11-01 22:00:00 +0000
tags: ["LLMs"]
category: "ai"
description: "Colibri runs GLM-5.2 on ordinary machines by keeping routed experts on disk. I read the source to see how routing, caching and pinning work, and where the claims need care."
keywords: "colibri llm engine, run llm from disk, mixture of experts inference, GLM-5.2 local, expert caching, int4 quantization, speculative decoding, MoE routing"
image: /assets/images/og-colibri-llm-engine-explained.png
---

I wanted to know how [Colibri](https://github.com/JustVugg/colibri), an open-source C engine, can run a 744B model on an ordinary machine. So I read the source at commit [`0a5504a`](https://github.com/JustVugg/colibri/tree/0a5504a).

I did not run it. Every speed below is the project's own figure, unless I say it is my estimate.

## What is the core idea?

A mixture-of-experts (MoE) model is huge on disk, but each token uses only a small part of it. So Colibri doesn't need the whole model in fast memory.

Attention, embeddings, routers and the other dense parts stay in RAM. The routed experts stay on disk. They are read when the router picks them, and then cached.

## How big is GLM-5.2, really?

These numbers come from the [model's config](https://huggingface.co/zai-org/GLM-5.2/raw/main/config.json): 78 layers with the first 3 dense, 256 experts per layer, top-8 routing, hidden size 6144 and expert width 2048.

| Quantity | Calculation | Result |
|---|---|---|
| One expert | 3 × 6144 × 2048 | 37.7M parameters |
| All routed experts | 75 × 256 = 19,200 experts | 725B of the 744B parameters |
| Experts used per token | 75 × 8 = 600 experts | 22.6B parameters |
| One expert on disk (int4 + scales) | 18 + 2.25 MiB | 21.2 MB |
| Worst-case read per token | 600 × 21.2 MB | about 12.7 GB |

The 21.2 MB matches [a comment in the code](https://github.com/JustVugg/colibri/blob/0a5504a/c/colibri.c#L12853). The README says about 11 GB per token. That number fits the per-row format, which I work out at about 11.35 GB, not the current one.

{% include fig.html src="colibri-llm-engine-explained/model-split.svg" alt="Bars comparing 724.8B parameters in all routed experts, 19.2B in everything else and 22.6B in the experts used per token. One expert is 21.2 MB on disk and the worst-case read is 12.7 GB per token." caption="Each token touches about 3% of the routed experts." %}

## What happens at each layer?

All of this is in [`moe()`](https://github.com/JustVugg/colibri/blob/0a5504a/c/colibri.c#L6731).

1. **Route.** A float32 router scores all 256 experts with a sigmoid and ranks them by score plus bias. The chosen experts are then weighted by the unbiased score, normalised and multiplied by 2.5.
2. **Union.** When several positions run together, each expert loads only once, even if many positions chose it.
3. **Place.** Each expert is looked up in [the pinned set, then the per-layer LRU cache, then disk](https://github.com/JustVugg/colibri/blob/0a5504a/c/colibri.c#L7209).
4. **Compute.** Each expert computes [`down(SiLU(gate(x)) * up(x))`](https://github.com/JustVugg/colibri/blob/0a5504a/c/colibri.c#L1840). The results are added back with their weights, and the shared expert is added separately.
5. **Learn.** The engine updates its usage, heat and recency counters. This is only bookkeeping about which experts get used. No weights are trained.

{% include fig.html src="colibri-llm-engine-explained/expert-lookup.svg" alt="Three boxes joined by arrows: pinned set in RAM, then LRU cache in RAM, then a 21.2 MB read from disk." caption="Only an expert missing from both RAM tiers costs a disk read." %}

## What sets the speed?

My rough estimate is this:

$$
\text{time per token} \approx \frac{\text{miss share} \times 12.7\ \text{GB}}{\text{disk speed}} + \text{compute}
$$

I assumed a disk at 7 GB/s and left compute out, so this curve is only a ceiling.

{% include fig.html src="colibri-llm-engine-explained/speed-ceiling.svg" alt="Curve of the maximum tokens per second against the share of experts missing, at 7 GB per second. It is 2.20 at 25 percent missing and 0.55 at 100 percent." caption="If every expert misses, reading alone takes about 1.8 s per token, so 0.55 tok/s at most." %}

The project's numbers fit that picture. A cold laptop with 25 GB of RAM gets 0.05 to 0.1 tok/s. A [32-core EPYC with a 97% hit rate](https://github.com/JustVugg/colibri/blob/0a5504a/docs/benchmarks.md) gets 2.34 tok/s, and its logs show the CPU as the limit, not the disk.

This is also why a GPU helps differently from model to model. It is mainly a faster place to hold experts and dense layers. Experts it doesn't hold still run on the CPU.

## How does it "learn"?

The README says Colibri gets faster the more you use it. Here is what that means in the code.

`.coli_usage` stores how often each layer and expert pair was selected, and it's saved every turn. At startup, experts are [ranked across all layers by count](https://github.com/JustVugg/colibri/blob/0a5504a/c/colibri.c#L12845) and pinned until the budget runs out. Auto-pinning [starts at 5,000 selections](https://github.com/JustVugg/colibri/blob/0a5504a/c/colibri.c#L14921) and grows to half the expert budget by 200,000.

{% include fig.html src="colibri-llm-engine-explained/pin-ramp.svg" alt="Line showing the pinned share of the expert budget. It is zero below 5,000 selections, then grows to 50 percent at 200,000 selections and stays there." caption="The pinned share grows with the history, and stops at half the budget." %}

So the gain mostly shows up across restarts. Within a session, only the LRU cache adapts, unless you turn on `--repin`.

## What other techniques does it use?

**Int4 weights.** Weights are stored as 4-bit values, with a float32 scale for every 64 inputs. The router and norms stay float32.

**Compressed KV cache.** GLM-5.2's attention stores 576 values per token per layer instead of 32,768. I computed the sizes at 131,072 tokens. They are close to the README's 23.9 and 11.9 GB for GLM-5.3.

{% include fig.html src="colibri-llm-engine-explained/kv-size.svg" alt="Bars of KV cache size at 131,072 tokens: 1,340 GB for 32,768 floats in float32, 23.6 GB for 576 floats in float32 and 11.8 GB in float16." caption="576 floats instead of 32,768 makes the cache 57 times smaller." %}

**Saved conversations.** The KV cache is [saved to disk by default](https://github.com/JustVugg/colibri/blob/0a5504a/c/colibri.c#L11064), so a conversation can reopen without rereading the prompt.

**Speculative drafts.** A small extra head (MTP) drafts tokens, and the main model [checks them in one batch](https://github.com/JustVugg/colibri/blob/0a5504a/c/colibri.c#L9330). Only accepted tokens advance the KV cache. The project reports 2.2 to 2.8 tokens per forward pass when it pays off.

**Prefetch (`PILOT`).** This runs the next layer's router on the current state, to guess which experts to load. The project [reports 71.6% recall](https://github.com/JustVugg/colibri/blob/0a5504a/docs/tuning.md). It only loads ahead, and it's off by default.

## Where do the claims need care?

**"Placement only ever decides speed."** This holds on the default path, with some exceptions. [`CACHE_ROUTE`](https://github.com/JustVugg/colibri/blob/0a5504a/c/colibri.c#L6832) swaps in experts already in memory, and [`EXPERT_BUDGET`](https://github.com/JustVugg/colibri/blob/0a5504a/c/colibri.c#L6999) drops missing ones. [`DEGRADE_ZERO`](https://github.com/JustVugg/colibri/blob/0a5504a/c/colibri.c#L7095) zeroes low-weight ones, which the project measured at +2.9% perplexity at one setting and +41% at another. `TOPK` and `--topp` change the selection too. All of these are opt-in. The README also says a long answer on the GPU can drift from the CPU's by a word.

**"A faithful model."** This means faithful to the int4 file, not to the original weights. On OLMoE, [per-row int4 lost 8.2 points of accuracy](https://github.com/JustVugg/colibri/blob/0a5504a/docs/benchmarks.md), and grouped scales recover about 63% of that. By my arithmetic that leaves about 3 points. GLM-5.2 scored 62.5% with n = 40, and I found no full-precision baseline to compare it with. So I would treat its real quality cost as unmeasured.

**The docs and the code disagree on `PIPE`.** [`tuning.md`](https://github.com/JustVugg/colibri/blob/0a5504a/docs/tuning.md?plain=1#L22) says the async loader is on by default, but [the engine enables it only on Windows](https://github.com/JustVugg/colibri/blob/0a5504a/c/colibri.c#L14312). On Linux and macOS, disk loads block unless you set `PIPE=1`.

**Overlap is partial.** Even with `PIPE` on, the CPU computes experts in order and [waits at each miss](https://github.com/JustVugg/colibri/blob/0a5504a/c/colibri.c#L7523). So an early miss stalls the resident experts behind it.

## What did I not check?

I read routing, the cache lookup, the expert computation, draft checking, pin ranking and the defaults above. I only skimmed the GPU tier and the prefetch code, so treat those parts as lighter.

## Why do I care?

Colibri's `POST /v1/systemone` [copies TypeSafe's Jev API](https://github.com/JustVugg/colibri/blob/0a5504a/docs/systemone.md) and returns a probability for each allowed answer. I'm building nanoJev, a small Jev-style encoder, so this gives me a possible comparison point.
