vLLM Hyper-Optimizes Kimi K3 on NVIDIA GB300 for Blazing-FAST Inference

What you could do with it

A chatbot company could give users nearly twice as fast replies when people ask one question at a time ¹.
A developer running Kimi K3-based AI assistants could cut infrastructure costs by getting more done on the same 16-GPU machines ¹.
A researcher testing long-context AI agents could see response times improve by up to 1.8× on real-world workloads ¹.

Speed Is Everything

The open-source software library vLLM has pushed the performance of Kimi K3 to new heights by building a completely custom set of GPU instructions, or kernels, optimized for NVIDIA Blackwell chips. For AI models that answer one word at a time — a process called decoding — every microsecond matters. The result: over 840 tokens per second on a single query using 16 NVIDIA GB300 GPUs with DSpark (synthetic AL=6), representing a 1.5–2× throughput increase in several tested configurations.

The Agentic Approach

vLLM is also changing how it develops these optimizations. For the first time, AI agents helped write and tune the most complex parts of the code — custom GPU kernels, fusing multiple operations into one, and coordinating communication across many GPUs ¹. The team says this marks a shift in how performance work is organized: instead of humans reviewing every single line, they are building systems that can verify correctness automatically. "We need to change how we organize performance work," the team wrote in its October 2026 blog post .

What's Optimized

The optimized decode path replaces almost every standard operation with a custom kernel built using NVIDIA's CuTe DSL framework. Key techniques include:

Shape tuning: Different batch sizes use different kernels, so GPU resources stay fully occupied at every workload.
Overlapping compute and communication: Attention output is fused directly with its all-reduce operation and the following residual calculation, cutting down on data movement between GPUs.
Improved Mixture of Experts (MoE) scheduling: The router scores and expert computations are split across GPU ranks, reducing redundant work and allowing the shared expert to run in parallel on a separate stream.
Prefetching weights into cache: While GPUs handle communication, a side kernel preloads the weights needed for the next operation, avoiding memory bottlenecks common in small-batch inference.

The stock vLLM implementation for Kimi K3 uses 130 microseconds and 23 kernels per layer. The optimized version cuts that to 69 microseconds and just 13 kernels per layer ¹.

Why It Matters

Faster decoding means lower costs and better user experience for any service running Kimi K3. The team found that decode throughput increased by up to 1.5–2× across multiple tested configurations, and P90 interactivity improved by up to 1.8× on the AgentX workload ¹. vLLM plans to apply the same agentic optimization approach to other models and hardware in the future, hoping the community helps build out the verification and validation tools needed to ship fast but safe code.

How good is it

vLLM's optimized decode path runs at over 840 tokens per second at batch 1 on 16 NVIDIA GB300 GPUs with DSpark, which is 1.5–2× faster throughput than the stock vLLM version across several tested configurations ¹. Each layer now finishes in just 69 microseconds using only 13 custom kernels, down from 130 microseconds and 23 kernels in the original implementation . On the AgentX agentic workload, P90 interactivity (the point below which 90% of responses fall) improved by up to 1.8× . The tests were run and reported by the vLLM team itself .

Picture: vllm.ai

Corrections and updates

Corrected on 23 October 2026. An earlier version claimed the optimization set a new bar for AI inference speed generally, and described the baseline as running at about half the speed; the source specified the exact GPU and DSpark conditions, and reported a 1.5–2× throughput increase rather than a simple half-speed comparison.

Key facts

Over 840 tokens/second at batch 1 on 16 NVIDIA GB300 GPUs with DSpark
Kernel count per layer dropped from 23 to 13, and latency from 130μs to 69μs
Up to 1.8× higher P90 interactivity on the AgentX workload
vLLM used AI agents to help write and tune custom GPU kernels
Optimizations target NVIDIA Blackwell GPUs with batch sizes 1–64 and optional context parallelism