vLLM Hyper-Optimizes Kimi K3 on NVIDIA GB300 for Blazing-FAST Inference

Speed Is Everything

The open-source software library vLLM has pushed the performance of Kimi K3 to new heights by building a completely custom set of GPU instructions, or kernels, optimized for NVIDIA Blackwell chips. For AI models that answer one word at a time — a process called decoding — every microsecond matters. The result: over 840 tokens per second on a single query using 16 NVIDIA GB300 GPUs, compared to the standard vLLM setup, which runs at about half that speed.

The Agentic Approach

vLLM is also changing how it develops these optimizations. For the first time, AI agents helped write and tune the most complex parts of the code — custom GPU kernels, fusing multiple operations into one, and coordinating communication across many GPUs ¹. The team says this marks a shift in how performance work is organized: instead of humans reviewing every single line, they are building systems that can verify correctness automatically. "We need to change how we organize performance work," the team wrote in its October 2026 blog post .

What's Optimized

The optimized decode path replaces almost every standard operation with a custom kernel built using NVIDIA's CuTe DSL framework. Key techniques include:

Shape tuning: Different batch sizes use different kernels, so GPU resources stay fully occupied at every workload.
Overlapping compute and communication: Attention output is fused directly with its all-reduce operation and the following residual calculation, cutting down on data movement between GPUs.
Improved Mixture of Experts (MoE) scheduling: The router scores and expert computations are split across GPU ranks, reducing redundant work and allowing the shared expert to run in parallel on a separate stream.
Prefetching weights into cache: While GPUs handle communication, a side kernel preloads the weights needed for the next operation, avoiding memory bottlenecks common in small-batch inference.

The stock vLLM implementation for Kimi K3 uses 130 microseconds and 23 kernels per layer. The optimized version cuts that to 69 microseconds and just 13 kernels per layer ¹.

Why It Matters

Faster decoding means lower costs and better user experience for any service running Kimi K3. The team found that decode throughput increased by up to 1.5–2× across multiple tested configurations, and P90 interactivity improved by up to 1.8× on the AgentX workload ¹. vLLM plans to apply the same agentic optimization approach to other models and hardware in the future, hoping the community helps build out the verification and validation tools needed to ship fast but safe code.

Key facts

Over 840 tokens/second at batch 1 on 16 NVIDIA GB300 GPUs with DSpark
Kernel count per layer dropped from 23 to 13, and latency from 130μs to 69μs
Up to 1.8× higher P90 interactivity on the AgentX workload
vLLM used AI agents to help write and tune custom GPU kernels
Optimizations target NVIDIA Blackwell GPUs with batch sizes 1–64 and optional context parallelism