Claude Code Leak Part II: Running Frontier AI Agents on Your Own Infrastructure

Why Run It Yourself?

Three reasons keep coming up in every conversation we have with engineering teams about this:

Data sovereignty. An AI coding agent sees everything—your source code, your environment variables, your database schemas, your business logic. For regulated industries or teams with sensitive IP, sending that through a third-party API is a non-starter. Running the model on your own infrastructure means your code never leaves your network.

Latency control. An agent making 20-30 tool calls per task needs fast round-trips. When the model is on your local network or the same cloud VPC, you eliminate the API latency that adds up across those calls.

Cost at scale. Claude Code subscriptions range from $100/month (Max 5x) to $200/month (Max 20x) per developer. For a team of 10, that's $1,000-2,000/month at subscription rates alone. But heavy agent usage can blow past subscription limits, pushing into pay-as-you-go API pricing (Opus at $5/$25 per million input/output tokens). Teams running intensive agentic workflows regularly see overages that push total costs to $3,000-5,000+/month. At that volume, the infrastructure cost of running your own model starts to look very attractive.

The Model Landscape: It's Not a One-Horse Race

Six months ago, DeepSeek-V3 was the obvious choice for self-hosted agentic workloads. That's no longer true. The open-weight model landscape has moved fast, and several models now compete for the top spot on the benchmarks that matter for coding agents.

SWE-bench Verified (Open-Weight Models, March 2026)

Not every "open" model is equally self-hostable. Some have downloadable weights you can run on your own GPUs. Others offer open benchmarks but are only available through cloud APIs. For an article about running agents on your own infrastructure, we only care about models you can actually download and serve privately.

For the rest of this article, we focus on the three models you can unambiguously download and run on your own hardware: DeepSeek V3.2, Kimi K2.5, and Qwen3-Coder-Next.

What Actually Matters for Self-Hosting: The Architecture

Benchmark scores are one dimension. For self-hosted agentic workloads, three architectural features determine how practical a model is to serve:

Mixture of Experts (MoE). All the top models use MoE now. The key numbers are total parameters (determines VRAM for weights) versus active parameters (determines compute per token). Kimi K2.5 has roughly 1T total. DeepSeek V3.2 activates 37B of 671B. Qwen3-Coder-Next activates just 3B of 80B. Smaller total means fewer GPUs needed. Smaller active means faster tokens per second.

Multi-head Latent Attention (MLA). This is the KV cache killer. Instead of storing full Key and Value projections for every token, MLA compresses them into a latent vector, reducing KV cache by roughly 5-10x. At 64K context, a standard model might consume 20 GB of VRAM for the KV cache of a single session. A model with MLA uses a fraction of that. For agent workloads where context fills up fast with tool outputs, this is a massive advantage.

Not every model has MLA. Here's how they differ:

• DeepSeek V3.2 and Kimi K2.5 both use MLA—they share the same attention infrastructure in vLLM and get the same KV cache compression benefits.
• Qwen3-Coder-Next uses a hybrid GatedDeltaNet architecture—a different approach to efficient attention, not MLA. Standard KV cache cost.

vLLM optimization depth. All three self-hostable models have dedicated implementations in vLLM with pipeline parallelism, MoE support, and speculative decoding. But the depth of optimization varies:

The Practical Takeaway

If you're running on Blackwell GPUs (g7e instances), DeepSeek V3.2 and Kimi K2.5 both get the MLA + Blackwell kernel path—the most refined serving option in vLLM. Kimi scores higher on SWE-bench but needs more VRAM (~500 GB Q4 vs ~350 GB). DeepSeek is the more VRAM-efficient choice with nearly as strong benchmarks.

If you want to minimize cost, Qwen3-Coder-Next (80B total, Q4 ~40 GB) fits on a single 96 GB GPU—though with only 3B active parameters, it remains to be seen whether it can sustain the compound constraint adherence that the agentic prompt architecture demands.

The landscape is moving fast. The right answer depends on your hardware budget, your concurrency needs, and how much KV cache pressure your agent sessions generate.

The GPU Math

Three things compete for your GPU memory: model weights, KV cache, and serving overhead. Let's be concrete.

Model Weights

FP8 quantization on Blackwell GPUs is hardware-accelerated and produces better quality than Q4. But at 1 byte per parameter, FP8 for a 671B model still takes ~670 GB—nearly filling a g7e.48xlarge on its own. For larger frontier models, Q4 is the practical choice. For smaller models like Qwen3-Coder-Next, FP8 becomes viable on a single GPU and is worth the quality improvement.

KV Cache Per Session

This is where people get surprised. The KV cache grows with context length and concurrent users, and agent sessions are context-hungry:

• System prompt + tool definitions: ~4-5K tokens before the conversation starts
• Each file read: 1-4K tokens added
• Each grep/test output: 1-2K tokens
• A typical multi-step coding task: 20-40K tokens accumulated

Models with MLA (DeepSeek, Kimi) compress the KV cache dramatically—realistic estimates are ~8-15 GB per session at 32K context. Qwen3-Coder-Next uses hybrid attention without MLA, so it will use more KV cache per session at the same context length. However, with only 80B total parameters, its overall VRAM footprint is still small enough to fit on a single GPU. The MLA advantage matters most for the larger models serving multiple concurrent sessions.

The Instance Nobody Can Get (And What to Use Instead)

If you've ever tried to spin up an AWS p5.48xlarge (8x H100), you know the pain. Capacity is almost always reserved or sold out. Even with capacity reservations, lead times can be weeks. P4d instances aren't much better. For a tool that's supposed to be always-available for your development team, "maybe we can get an instance" isn't an answer.

A g7e.48xlarge running DeepSeek-V3 at Q4: weights take roughly 350 GB across 8 GPUs, leaving over 400 GB for KV cache. With MLA compression, that's enough for multiple concurrent developer sessions at 64K+ context each. FP8 (~670 GB) would also fit but leaves barely 100 GB for everything else—tight for multi-user serving.

Parallelism: The Decision That Changes Everything

When your model spans 8 GPUs, you need a parallelism strategy. This choice affects latency, throughput, and how well the system handles multiple users. There are three options, and most people pick the wrong one.

Tensor Parallelism (TP): Fast on NVLink, Painful on PCIe

Splits each layer across all GPUs. Every token generation requires an all-reduce synchronization across all 8 GPUs. With NVLink (H100 SXM), this is fast. With PCIe (which the g7e likely uses), this becomes a bottleneck. You end up GPU-compute-bound for microseconds, then waiting on PCIe for milliseconds. TP is widely used and well-documented, but it assumes high-bandwidth interconnects that inference-class instances don't always have.

Pipeline Parallelism (PP): The Practical Choice Before EP

Assigns different layers to different GPUs. Data flows sequentially: GPU 0 processes layers 0-15, passes the result to GPU 1 for layers 16-31, and so on. The inter-GPU communication is a single point-to-point activation transfer, not an all-to-all synchronization. PCIe handles that fine.

Before expert parallelism existed, PP was the practical answer for multi-GPU inference on PCIe hardware. The downside is single-request latency—each token must traverse all GPUs sequentially. But with vLLM's continuous batching and multiple concurrent users, the pipeline stays full. The more users, the better your GPU utilization. For non-MoE models, PP remains a strong choice on PCIe-connected instances.

Expert Parallelism (EP): What MoE Models Actually Want

This is the interesting one, and it's what we'd recommend for DeepSeek-V3 specifically. Instead of splitting layers or splitting each layer, EP splits experts across GPUs. DeepSeek-V3 has 256 experts. With 8 GPUs, each GPU holds 32 experts and only runs compute for the experts routed to it on each token.

What vLLM Can Actually Do Today

We went through vLLM's source code to see what's real versus what's theoretical. The findings were better than we expected.

All Major Models Are Supported—But Not Equally

Every model we discussed has a dedicated implementation in vLLM with pipeline parallelism, MoE integration, and speculative decoding. They're all production-viable. But the optimization depth varies.

DeepSeek V3.2 and Kimi K2.5 share the MLA attention infrastructure and get the deepest optimization path: dedicated fp8_ds_mla KV cache, 6 MLA attention backends, FlashMLA Sparse targeting Blackwell (SM100) GPUs, and DeepGEMM kernels with E8M0 precision. On Blackwell hardware, this is the most refined serving path in vLLM.

Qwen3-Coder-Next uses a hybrid GatedDeltaNet attention with full MoE support and its own speculative decoding method, but without the MLA-specific Blackwell kernel optimizations. It works well on vLLM—just without the KV cache compression advantage.

Expert Parallelism Infrastructure

vLLM has a mature EP stack. The --enable-ep-weight-filter flag means each GPU only loads its own experts during startup—a massive I/O win for 671B models. Expert placement strategies include linear and round_robin distribution. EPLB runs asynchronously with configurable windows (default 1000 tokens) and rebalancing intervals (default 3000 steps).

Seven all2all communication backends are available, including DeepEP kernels specifically optimized for MoE expert dispatch, and FlashInfer NVLink variants for both high-throughput and low-latency workloads.

Multi-User Serving

This is where vLLM earns its place in this stack. For agent workloads where multiple developers share a model:

• Chunked prefill is enabled by default. New users joining a session don't block ongoing token generation for other users.
• Continuous batching interleaves tokens from different users in each forward pass.
• CPU KV cache offloading is a new (experimental) feature that spills KV cache to CPU RAM when GPU VRAM fills—directly addressing the problem of multiple long-running agent sessions.
• Speculative decoding with Eagle3 uses a small draft model to propose tokens that the large model verifies in batch. For agent workloads where the model often generates predictable boilerplate (git commands, file paths, structured output), this significantly boosts tokens per second.

One notable gap: vLLM has no Neuron/Trainium backend. If you go the Trainium2 route, you'd need AWS's neuronx-distributed libraries, which lack vLLM's PagedAttention and continuous batching.

Putting It Together: Three Reference Configurations

There's no single right answer. Here are three setups we'd consider depending on team size and budget:

For Options A and B with multi-GPU EP: each GPU holds replicated attention weights plus its share of expert weights, with KV cache stored locally per GPU. The math depends on the model, but the pattern is the same—budget for weights plus KV cache per concurrent session, and don't exceed ~80-85% of GPU VRAM to leave headroom for serving overhead.

The Cost Reality Check

Let's not pretend this is cheap. A g7e.48xlarge is a serious instance. But compare it to the alternative:

There's also the engineering cost of running this. Someone has to manage the instance, monitor vLLM, handle updates, deal with model upgrades. If your team doesn't have MLOps capability, factor in the learning curve or the cost of bringing that expertise in.

The Honest Assessment

Can you run a Claude Code-style agentic system on private infrastructure today? Yes. Multiple open-weight models score above 70% on SWE-bench Verified. vLLM's MoE and MLA infrastructure is production-grade. And g7e Blackwell instances solve the availability problem that plagued H100s.

Is it as good as Claude Code running on Anthropic's infrastructure with their frontier model? Not yet. The gap is real, especially for the hardest tasks—multi-agent coordination, adversarial verification, sustained constraint adherence over long sessions. But for most daily coding tasks, a well-configured setup with DeepSeek V3.2 or Kimi K2.5 will serve your team well, with the added benefits of data sovereignty and cost predictability.

The smart play is to start small. Run Qwen3-Coder-Next on a single GPU. Test whether the agentic prompt architecture works with your codebase. If the model capability is the bottleneck, scale up to DeepSeek V3.2 or Kimi K2.5. If concurrency is the bottleneck, add GPUs. Don't buy 8 GPUs on day one.