Running Kimi K3 on CPU: A 2.8-Trillion-Parameter Model Without a Single GPU

The Hardware Problem

Kimi K3 was announced a few weeks ago, and Moonshot AI released the weights publicly on July 27th. Open weights, however, are not the same thing as accessible inference. The model is a mixture-of-experts (MoE) design with 2.8 trillion total parameters — the checkpoint alone is 1.56 TB — and the typical minimum GPU setup for serving it is a full node of eight NVIDIA B300 GPUs. As of this writing, renting that class of hardware runs $8–10 per GPU-hour on marketplace providers like RunPod — roughly $70–100 per hour for the node, and that's assuming you're lucky enough to actually get an eight-B300 allocation. On a hyperscaler like AWS, the equivalent costs considerably more.

But the MoE architecture is also what makes an alternative thinkable. K3 has 896 experts per layer and activates only 16 of them per token: each forward pass touches a small fraction of the total weights — about 26 GB of expert weights per generated token. You don't need all 2.8 trillion parameters in fast memory at once; you need the right slice of them, fast enough.

We found a project on GitHub called rabbit, a Rust CPU inference engine (itself a port of colibrì) built around exactly this idea: keep the dense part of the model resident in RAM, quantized, and stream the thousands of routed experts from disk on demand. It could already run Kimi K3 correctly — validated token-exact against Moonshot's own reference code — on a machine with 123 GB of RAM. Correctly, but slowly: decode on that machine ran at 50–70 seconds per token, because every token had to pull experts off an NVMe drive.

So we asked the obvious question: what if the disk wasn't in the loop at all? What if the entire expert set fit in RAM, on a machine with serious CPU power?

Step Zero: Understand the Architecture

Before optimizing anything, we used Claude Code to analyze the codebase and the model, and to build an interactive visualization of the Kimi K3 architecture — the layer stack, the per-layer dataflow, the internals of its two attention mechanisms (KDA and MLA) and its latent-width MoE, with every tensor shape derived from the real config. It even enumerates every matrix multiplication in the model and reconciles them exactly against the parameter count. You can explore it here.

That exercise paid for itself immediately: it told us where the compute actually goes. Per decoded token, K3 performs 1,472 expert applications (16 experts × 92 MoE layers), around 170 GFLOP of total compute, and streams ~26 GB of MXFP4-quantized expert weights through the memory system. Those three numbers shaped every decision that followed.

From 36.8 Seconds per Token to 1.26

The first honest baseline on the big machine was sobering: 36.8 seconds per token, plus 13 minutes just to load the model. Against the ~0.04 s/token bandwidth floor, roughly 99.9% of the machine's potential was being left on the table. Closing that gap was the work — and every step below was measured with a deterministic benchmark (the same token sequence every run) and verified bit-identical against the reference output, so speed never came at the cost of correctness.

1. Build a cheap way to iterate

A single experiment on the real 1.56 TB checkpoint costs 20 minutes, mostly loading. So the first thing built was a synthetic miniature checkpoint: real layer widths, only 6 layers, ~90 GB. It reproduced the real model's per-token behavior almost exactly and turned every subsequent experiment from hours into minutes.

2. A 5× faster kernel that changed nothing — and diagnosed everything

The hottest code in the engine is the matrix multiply that decodes K3's MXFP4-quantized expert weights, and it was plain scalar code. Rewriting it with AVX-512 vector instructions made the kernel itself 3–5× faster — and end-to-end generation got approximately 0% faster. That "failure" was the key diagnosis of the whole project: with 384 threads fighting over thousands of tiny sub-tasks per matrix multiply, decode was bound by thread-scheduling overhead, not arithmetic. The fast kernel was necessary, but something else was hiding it.

3. Restructure the parallelism

Instead of running each expert's math one after another — each fanning out across all cores and synchronizing, roughly 4,400 fork/join points per token — the engine now runs all 16 selected experts of a layer in parallel, in coarse row-block tasks sized to the machine. Combined with the kernel work, the real checkpoint went from 36.8 to 16.9 seconds per token, with the output provably unchanged, and prompt processing 2.3× faster as a side effect.

4. Treat the machine as what it is: six memory domains

With 3 TB of RAM, the full expert set fits in memory — but a 2-socket, 6-NUMA-domain box doesn't behave like one flat pool. By default all the expert weights piled up on whichever memory nodes happened to load them, and 384 threads then contended on one or two memory controllers (we measured cross-socket memory reads at less than half the speed of local ones). Three fixes, in order of increasing effort:

• Preload everything, in parallel. A new --preload-experts mode loads all 1.3 TB of experts up front, so the very first token runs at steady-state speed. Parallelizing the load cut warmup from ~30 minutes to ~9.
• Interleave memory pages across all six NUMA nodes — a zero-code numactl launch change worth 1.4× on its own.
• Give each expert a home socket. Each expert is loaded by, and later computed by, a thread pool pinned to the socket that owns its memory. Just as important: attention and expert math want opposite thread counts (attention degrades past ~48 threads while expert math scales past 192), so the two now run on separately sized pools.

5. Measure, keep, revert

Not everything survived contact with the profiler. Sharding the model's output-projection weight across sockets: kept — 2.7× on that step. Sharding the attention projections the same way: measured as a regression — the cost of waking a sleeping thread pool 138 times per token exceeded the memory-locality win — so it was built, measured, and reverted. The project's performance log records the failures alongside the wins, deliberately: a technique that fails with a number attached is one nobody wastes a week re-trying.

The Scoreboard

All numbers below are from the real 2.8T checkpoint on the same machine. The first two rows are a controlled 12-token benchmark; the last two are live serving measurements (41-token prompt, 128-token completions), where prompt processing is folded into the serving turn.

• Load time is the fixed cost of loading the dense, always-resident part of the model. Preload is the optional up-front load of all 896 experts × 92 layers (~1.3 TB) into RAM; without it, the first few hundred tokens run slower while the cache warms.
• Every step produced bit-identical output to the baseline, verified by fingerprinting the full logits of every generated token.

And the headroom is mapped, not hypothetical: the measured per-socket compute floor is ~0.29 s/token at current pool widths, the binding constraint (thread-pool wake latency on high-frequency fan-outs) is diagnosed with numbers attached, and the AMX matrix units this CPU ships — purpose-built for exactly this kind of batched quantized math — are still entirely unused.

The Cost Reality Check

Why This Matters

Let's be candid about what this project was: an experiment, driven by curiosity. We never expected the result to be immediately useful — we simply wanted to know what kind of speed $27/hour of ordinary CPU could buy against a 2.8-trillion-parameter frontier model. The answer — about 0.8 tokens per second, with a measured path to several times that — turned out to be far more interesting than we expected.

These numbers are still far from production-grade inference throughput — 1.26 seconds per token won't serve a chatbot to a thousand users. But that's not really the point. The software gap to this machine's ~0.04 s/token hardware floor is now well understood, the remaining levers are identified and measured, and hardware only gets cheaper.

And even today's speed is enough for a class of real work — the kind where privacy outranks latency. Overnight batch jobs: summarizing, classifying, or extracting from documents that cannot leave your network. Evaluating a frontier open-weight model against your own sensitive data before committing to GPU infrastructure. Low-volume, high-stakes generation — contracts, medical records, incident reports — where nobody is waiting on a chat cursor. None of that needs fifty tokens per second.

A 2.8-trillion-parameter frontier model running entirely on a CPU server — no GPUs, no cloud API, no per-token bill, no data leaving your network — is a genuinely interesting glimpse of the future of private LLMs, and of the democratization of AI as a whole. The weights are open. Increasingly, the hardware to run them is just... a server.