1. Running Kimi K3 on GPU
K3 is big: the model file alone is 1.56 TB. As the CPU post explained, the usual minimum to serve it is one server with eight of NVIDIA's top-end B300 GPUs. We used the previous generation, the B200, which has less memory per GPU, so one server of eight isn't enough. We used two.
The software side is standard. vLLM, the widely used open-source serving engine, runs the model. The vLLM recipes site has the setup details. In front of it sits a small gateway (LiteLLM), so every request from the Kimi agent passes through one place we control. When something breaks, it fails loudly instead of quietly retrying, because we'd rather a human take a look.
One thing we learned: AI agents have far longer conversations than chat users. They read files, run commands, read the results, and keep going for hours. Ours filled the model's entire working memory and had to summarize itself to continue. If you're sizing GPUs for agents, size them for long sessions.
1.1 Hardware Configuration
- Servers: 2 GPU instances
- GPUs: 8× NVIDIA B200 per instance (16 in total)
- GPU memory: roughly 3 TB in total, enough for the 1.56 TB model plus room for long agent sessions
- Software: vLLM (setup at recipes.vllm.ai), with a LiteLLM gateway in front
- Cost: approx. $150/hour
- Speed: 200–300 tokens/sec per conversation
2. Limitations of Closed-Source Frontier Models
We're not purists. For the CPU project we used Claude Code, a closed model, to study a public codebase, and we'd do it again. The question isn't open versus closed. It's what data the model gets to see. Four things pushed us to run our own model for the main agent.
Less data leaves. Security logs from a company's computers contain machine names, user names and command histories. Together, that's a map of the network. Our main agent had to read them to do the job. With a hosted service, that data lands in someone else's systems under someone else's retention rules. On GPU servers we rent and run ourselves, it stays on infrastructure we control. This engagement wasn't fully self-hosted: the two reviewing agents ran on Claude, a hosted model, and saw client data too. Running the main agent on our own servers reduced what left. It didn't make it zero.
You don't decide where the line is. Security testing means asking a model to reason about how attackers break in, move around and cover their tracks. Commercial providers have usage policies, and they're right to. But where they draw the line, and whether it moves next month, isn't up to you. A test that half-works because some requests were quietly toned down is worse than one that fails outright, because you can't tell which half you got. To be clear, we didn't compare the two head to head. It's a risk we chose not to take, not a failure we saw. Running open weights means the line is yours, and so is the job of building the guardrails.
Repeatability. Every result in this engagement was frozen and fingerprinted so it can be checked later. A hosted model can be updated under you without notice, and you can't re-run last month's work on last month's model. Self-hosted weights don't change unless you change them.
Cost under agent load. An agent working all day, rereading the same files. Per-use pricing grows with all of that, while a rented GPU server costs the same per hour no matter how hard you push it. The catch: it also costs the same when it sits idle.
What About Where the Model Comes From?
Some companies don't trust where certain open models come from, and won't use them. Much of that worry is about data: call the maker's hosted service and your prompts land on their servers, under their rules. Self-hosting removes that path. The only thing we took from the model's maker was the weights; vLLM's own open-source code ran them, on rented GPU servers with no internet access, so nothing could be sent back to the maker. What self-hosting doesn't remove is whatever was trained into the weights. You can't inspect 2.8 trillion parameters for hidden behavior. That's one more reason our setup never trusted the model: its fake attacks can't run, agents from a different vendor checked its output, and a human had the final say.
What We Gave Up
The very best closed models are still generally stronger at long, multi-step agent work; someone has to run the servers; and the provider's safety work becomes your safety work.
3. Red Teaming / White-Hat Hacking with Kimi K3
The Problem
The client runs an attack-detection system over security logs from two sites. The question: does it catch attacks without flagging everything else? You can't answer that without attacks, and real ones don't show up on demand. Months of logs held exactly one known red-team exercise. So we built synthetic ones: fake log entries that look like an attack to the detector. None of it can actually run. It can only be scored.
Kimi K3's job was to write the software that generates these fake attacks and to keep them realistic: placed on realistic machines, at realistic times, mixed in with everyday activity. Otherwise the test means nothing.
Three Agents, One Human
We ran three AI agents side by side, each with one job. A Kimi K3 agent built the fake attacks. Two Claude agents scored them against the detector and checked every record for mistakes. A human (us) sat above all three. The rules:
- Nothing goes into the shared dataset unless both other agents approve and the human says go.
- If one agent says something about another agent's work, it has to point to the file or message. Otherwise it's a question, not a fact.
What Went Wrong
As in the CPU project, we wrote down the failures alongside the wins, on purpose. Kimi K3 wrote most of the generator, and most of the mistakes were its own. Every one was caught, either by another agent or by the human.
- It overwrote work the others had already checked. It re-ran the generator into the same folder to fix one small file, and everything in the folder changed. The old version was gone.
- Same input, different output. A reviewing agent found why: a database query returned results in a different order on each run, so the "same" setup produced different fake attacks every time. The fix was small. All three agents then confirmed identical output on repeated runs.
- It stated things it hadn't checked. It told another agent that its code had a bug. It didn't, and our agent hadn't read the code. The rule that came out of it: quote the line you read, or ask.
- It ran once more after being told to stop. "Just one more identical run" is still not stopping.
- It wrote to the shared dataset without permission. The file also contained real account names. It was deleted.
What fixed it was process, not the model
- Run everything twice and check you get the same answer before asking anyone else to review it.
- Never overwrite something someone has already checked. Always write to a new place.
- Freeze final results with a fingerprint, and have someone else verify it.
- A human approves every change to shared data.
The final set of fake attacks was frozen, fingerprinted and independently verified by the other two agents. It's now ready to serve as the yardstick for the client's detector.
Why This Matters
The CPU post ended with a list of work where privacy matters more than speed, including “evaluating a frontier open-weight model against your own sensitive data.” This engagement is exactly that, at GPU speed: a 2.8-trillion-parameter open model, on hardware we control, doing the core of a real security job, with closed-model agents reviewing its work.
The bigger lesson is less glamorous: the model is not the safety mechanism. Kimi K3 was capable, and it also made the classic AI-agent mistakes: confident claims it hadn't checked, a destructive rerun, one action after a stop. What made the final result trustworthy was everything around the model: harmless-by-design test data, a second agent checking the first, fingerprinted results, and a human saying go. That's true whichever model you use, open or closed.
Interested in testing your detection stack?
If your team wants to know whether your detection system actually catches attacks, and wants a say in which models see your logs, get in touch.