vLLM v0.18.0 — Release Notes & Engineering Impact

# vLLM v0.18.0 ## Known issues - Degraded accuracy when serving Qwen3.5 with FP8 KV cache on B200 (#37618) - If you previously ran into `CUBLAS_STATUS_INVALID_VALUE` and had to use a workaround in `v0.17.0`, you can reinstall `torch 2.10.0`. PyTorch published an updated wheel that addresses this bug. ## Highlights This release features 445 commits from 213 contributors (61 new)! * **gRPC Serving Support**: vLLM now supports gRPC serving via the new `--grpc` flag (#36169), enabling high-performance RPC-based serving alongside the existing HTTP/REST interface. * **GPU-less Render Serving**: New `vllm launch render` command (#36166, #34551) enables GPU-less preprocessing and rendering, allowing separation of multimodal preprocessing from GPU inference. * **NGram GPU Speculative Decoding**: NGram speculative decoding now runs on GPU and is compatible with the async scheduler (#29184), significantly reducing spec decode overhead. * **KV Cache Offloading Improvements**: Smart CPU offloading that stores only frequently-reused blocks (#35342), plus FlexKV as a new offloading backend (#34328) and support for multiple KV groups in offloading spec (#36610). * **Elastic Expert Parallelism Milestone 2**: NIXL-EP integration (#35627) enables dynamic GPU scaling for MoE experts, with new `--enable-ep-weight-filter` CLI option (#37351) for faster EP model loading. * **FlashInfer 0.6.6**: Updated FlashInfer dependency (#36768) with numerous performance and correctness improvements. * **Responses API Streaming Tool Calls**: The OpenAI Responses API now supports tool/function calling with streaming (#29947). * **Online Beam Search for ASR**: Beam search support for encoder/decoder models both offline (#36153) and online transcriptions (#36160). * **Ray No Longer a Default Dependency**: Ray has been removed as a default dependency (#36170) — install it explicitly if needed. ### Model Support * **New architectures**: Sarvam MoE (#33942), OLMo Hybrid (#32550), HyperCLOVAX-SEED-Think-32B VLM (#31471), HyperCLOVAX-SEED-Think-14B (#37107), Kimi-Audio-7B-Instruct (#36127), ColPali late-interaction retrieval (#36818), ERNIE pooling models (#36385). * **Speculative decoding**: Eagle3 for Qwen3.5 (#36658), Eagle3 for Kimi K2.5 MLA (#36361), Eagle for Mistral Large 3 with dense layers (#36163). * **LoRA**: Whisper LoRA (#29856), FP8 LoRA dense kernel (#35242). * **Multimodal**: Online use_audio_in_video (#36319), audio extraction from MP4 for Nemotron Nano VL (#35539), audio transcription for MP4/M4A/WebM (#35109), expose media_io_kwargs at runtime (#34778), fast media preprocessing for Nano Nemotron VL (#35657). * **Compatibility**: Gemma/Gemma2 inputs_embeds (#36787), SigLIP/CLIP Transformers v5 (#37200), fused expert weights in Transformers backend (#36997). * **Performance**: Qwen3 Next fused GDN kernel (#35777), LFM2 tuned H100 MoE configs (#36699). * **Fixes**: DeepSeek-V3.2 tokenizer space stripping (#37004), Qwen3.5 tool calling (#36774), Qwen3-VL timestamp mismatch (#36136), Qwen3-Next TP>1 weight sharding (#36242), Qwen3-ASR torch.compile (#35869), MiniCPM-V audio inference (#36751), MiniCPM-O 4.5 ViT attention (#34127), routed experts for hybrid models (#35744), Qwen2.5-Omni/Qwen3-Omni multi-video audio_in_video (#37147), DeepSeek-OCR empty images crash (#36670). ### Engine Core * **Model Runner V2**: Probabilistic rejection sampling for spec decode (#35461), pooling models (#36019), extensible CUDA graph dispatch (#35959), WhisperModelState (#35790), XD-RoPE (#36817), model_state CUDA graph capture (#36544). * **KV cache offloading**: Reuse-frequency-gated CPU stores (#35342), FlexKV offloading backend (#34328), multiple KV groups (#36610), async scheduling fix (#33881). * **Speculative decoding**: NGram GPU implementation with async scheduler (#29184), fused EAGLE step slot mapping (#33503). * **Performance**: Remove busy loop from idle buffer readers (#28053), 2.7% E2E throughput for pooling via worker-side maxsim (#36159), 3.2% via batched maxsim (#36710), CUDA graph memory accounting during profiling (#30515), checkpoint prefetch to OS page cache (#36012), InstantTensor weight loader (#36139), sporadic stall fix via pin_memory removal (#37006). * **Stability**: VLM concurrent throughput degradation fix (#36557), DP deadlock fix (#35194), DeepSeek V3.2 OOM during CG profiling (#36691), Ray DP startup crash (#36665), NCCL rank calculation fix (#36940), zero-init MLA output buffers for NaN prevention (#37442), CUDA OOM fix (#35594). * **Defaults**: Cascade attention disabled by default (#36318). * **Extensibility**: OOT linear method registration (#35981), custom collective ops registration for non-CUDA platforms (#34760). ### Kernel * **FA4 for MLA prefill** (#34732). * **FlashInfer Sparse MLA**: FP8 KV cache support (#35891), CUDA graphs on ROCm (#35719), MTP lens > 1 on ROCm (#36681). * **TRTLLM FP8 MoE modular kernel** (#36307). * **FP8 KV cache for Triton MLA decode** (#34597). * **FlashInfer MoE A2A kernel** (#36022). * **Remove chunking from FusedMoE** for full batch processing (#34086). * **CustomOp FusedRMSNormGated** for torch.compile compatibility (#35877). * **Mamba2 SSD prefill Triton kernel** optimization (#35397). * **DeepSeek-V3.2**: Vectorized MLA query concat kernel (#34917), optimized FP8 KV cache gather for context parallel (#35290). * **320-dimension MLA head size** support (#36161). * **Packed recurrent fast path** for decode (#36596). * **EP scatter race condition** fix (#34991). ### Hardware & Performance * **NVIDIA**: FA4 for MLA prefill (#34732), DeepSeek-V3.2 MLA kernel optimizations (#34917, #35290). * **AMD ROCm**: Sparse MLA CUDA graphs (#35719), MTP lens > 1 in Sparse MLA (#36681), MLA with nhead<16 + FP8 KV for TP=8 (#35850), RoPE+KV cache fusion for AITER FA (#35786), AITER MLA CPU sync avoidance (#35765), Quark W4A8 MXFP4/FP8 (#35316), gfx1152/gfx1153 Krackan support (#36499), fused_topk_bias AITER optimization (#36253), skinny GEMM improvements (#34304), DeepEP in ROCm Dockerfile (#36086), startup OOM fix (#36720). * **Intel XPU**: Model Runner V2 enabled (#36078), MLA Sparse backend for DeepSeek V3.2 (#33230), LoRA via torch.compile (#36962), block FP8 MoE fallback (#36458), deepseek_scaling_rope fused kernel (#36612). * **CPU**: aarch64 int8 matmul via OneDNN upgrade (#36147), AMD Zen CPU backend via zentorch (#35970). * **RISC-V**: CPU backend support (#36578). * **Performance**: 5% E2E improvement for PD disaggregation scheduling (#35781), packed recurrent decode fast path (#36596), pooling model maxsim 2.7%+3.2% throughput (#36159, #36710). * **torch.compile**: FakeTensors instead of real GPU tensors for single-size compilation (#36093), non-contiguous fused RMSNorm + group quant (#36551), stop lazy compiling (#35472). ### Large Scale Serving * **Elastic EP Milestone 2**: NIXL-EP integration (#35627), `--enable-ep-weight-filter` for faster EP loading (#37351). * **PD Disaggregation**: ~5% scheduler overhead reduction (#35781), KV transfer fix with spec decode (#35158), P/D for hybrid SSM-FA models via NIXL (#36687), PP for multimodal models on Transformers backend (#37057). * **KV Connectors**: HMA + NIXL connector (#35758), FlexKV offloading (#34328), worker→scheduler metadata (#31964), All-to-All DCP backend (#34883). * **LMCache**: Fault tolerance mechanism (#36586), memory leak fix (#35931), race condition fix (#35831), TP size for MLA multi-reader locking (#36129). * **EP loading**: Skip non-local expert weights (#37136). ### Quantization * **ModelOpt MXFP8 MoE** support (#35986). * **MXFP4 MoE routing simulation** override for accuracy (#33595). * **FP8 LoRA dense kernel** (#35242). * **ROCm**: Quark W4A8 MXFP4/FP8 for LinearLayer (#35316), compressed-tensors fix for DeepSeek-R1 on MI300x (#36247). * **Fixes**: MLA crash with AWQ/GPTQ quantized models (#34695), score layer quantization for reranker models (#35849), GLM-4.1V non-default quantization (#36321), FP8 k_scale/v_scale loading for Qwen3-MoE (#35656). ### API & Frontend * **gRPC**: New `--grpc` flag for gRPC serving (#36169). * **GPU-less serving**: `vllm launch render` for preprocessing-only serving (#36166), `vllm launch` for GPU-less preprocessing (#34551). * **Responses API**: Streaming tool/function calling (#29947), reasoning item fixes (#34499, #36516). * **Anthropic API**: Accept redacted thinking blocks (#36992). * **ASR**: Online beam search transcriptions (#36160), offline beam search (#36153), audio transcription for MP4/M4A/WebM (#35109), realtime endpoint metrics (#35500). * **Tool calling**: Granite4 tool parser (#36827), Qwen3Coder anyOf double encoding fix (#36032). * **New options**: `--distributed-timeout-seconds` (#36047), `--attention-backend auto` (#35738), `reasoning_effort=none` (#36238), PyTorch profiler schedule (#35240). * **Cohere Embed v2 API** support (#37074). * **Azure Blob Storage** support for RunAI Model Streamer (#34614). * **Graceful shutdown** timeout for in-flight requests (#36666). * **Fixes**: tool_choice=required exceeding max_tokens crash (#36841), negative max_tokens with long prompts (#36789), concurrent classify/token_classify race (#36614), Anthropic billing header prefix cache miss (#36829), render endpoint crash for multimodal requests (#35684), xgrammar dtype mismatch on macOS CPU (#32384), minimax_m2 tool parser with stream interval > 1 (#35895). ### Security * Respect user `trust_remote_code` setting in NemotronVL and KimiK25 (#36192). * Upgrade xgrammar for security fix (#36168). * Guard RLHF weight sync deserialization behind insecure serialization flag (#35928). ### Dependencies * **FlashInfer 0.6.6** (#36768). * **Ray removed from default dependencies** (#36170). * `kaldi_native_fbank` made optional (#35996). * OpenAI dependency bounded to 2.24.0 (#36471). * Deprecated items from v0.18 removed (#36470, #36006). * Mistral common v10 (#36971). ### Breaking Changes 1. **Ray no longer a default dependency** — install explicitly if needed (#36170). 2. **Deprecated items removed** — items deprecated in v0.18 have been removed (#36470, #36006). 3. **Cascade attention disabled by default** (#36318). 4. **swap_space parameter removed** (V0 deprecation, #36216). 5. **Monolithic TRTLLM MoE disabled for renormalize routing** — late fix cherry-picked (#37591). ## New Contributors 🎉 * @11happy made their first contribution in https://github.com/vllm-project/vllm/pull/35481 * @12010486 made their first contribution in https://github.com/vllm-project/vllm/pull/36782 * @abhishkh made their first contribution in https://github.com/vllm-project/vllm/pull/32454 * @AjAnubolu made their first contribution in https://github.com/vllm-project/vllm/pull/35976 * @alvinttang made their first contribution in https://github.com/vllm-project/vllm/pull/36397 * @amd-asalykov made their first contribution in https://github.com/vllm-project/vllm/pull/35093 * @amd-lalithnc made their first contribution in https://github.com/vllm-project/vllm/pull/35970 * @arlo-scitix made their first contribution in https://github.com/vllm-project/vllm/pull/36139 * @benenzhu made their first contribution in https://github.com/vllm-project/vllm/pull/36253 * @ChuanLi1101 made their first contribution in https://github.com/vllm-project/vllm/pull/35893 * @cluster2600 made their first contribution in https://github.com/vllm-project/vllm/pull/34882 * @cong-or made their first contribution in https://github.com/vllm-project/vllm/pull/36164 * @daje0601 made their first contribution in https://github.com/vllm-project/vllm/pull/29856 * @davzaman made their first contribution in https://github.com/vllm-project/vllm/pull/32441 * @eellison made their first contribution in https://github.com/vllm-project/vllm/pull/35877 * @fangyuchu made their first contribution in https://github.com/vllm-project/vllm/pull/35194 * @feiqiangs made their first contribution in https://github.com/vllm-project/vllm/pull/34328 * @fenypatel99 made their first contribution in https://github.com/vllm-project/vllm/pull/35240 * @gambletan made their first contribution in https://github.com/vllm-project/vllm/pull/36402 * @giulio-leone made their first contribution in https://github.com/vllm-project/vllm/pull/36937 * @gkswns0531 made their first contribution in https://github.com/vllm-project/vllm/pull/35849 * @grimulkan made their first contribution in https://github.com/vllm-project/vllm/pull/34597 * @hai-meh-cs made their first contribution in https://github.com/vllm-project/vllm/pull/36684 * @hasethuraman made their first contribution in https://github.com/vllm-project/vllm/pull/34614 * @Hongbin10 made their first contribution in https://github.com/vllm-project/vllm/pull/36713 * @jeonsworld made their first contribution in https://github.com/vllm-project/vllm/pull/34499 * @jjmiao1 made their first contribution in https://github.com/vllm-project/vllm/pull/35994 * @Kaonael made their first contribution in https://github.com/vllm-project/vllm/pull/36818 * @ketyi made their first contribution in https://github.com/vllm-project/vllm/pull/36670 * @KevinZonda made their first contribution in https://github.com/vllm-project/vllm/pull/36209 * @leo-cf-tian made their first contribution in https://github.com/vllm-project/vllm/pull/36022 * @lisperz made their first contribution in https://github.com/vllm-project/vllm/pull/34531 * @mitre88 made their first contribution in https://github.com/vllm-project/vllm/pull/35933 * @nkm-meta made their first contribution in https://github.com/vllm-project/vllm/pull/34760 * @nvnbagrov made their first contribution in https://github.com/vllm-project/vllm/pull/35657 * @rahul-sarvam made their first contribution in https://github.com/vllm-project/vllm/pull/33942 * @royyhuang made their first contribution in https://github.com/vllm-project/vllm/pull/35931 * @sbeurnier made their first contribution in https://github.com/vllm-project/vllm/pull/37006 * @seanmamasde made their first contribution in https://github.com/vllm-project/vllm/pull/35109 * @sergey-zinchenko made their first contribution in https://github.com/vllm-project/vllm/pull/35684 * @shaunkotek made their first contribution in https://github.com/vllm-project/vllm/pull/36149 * @shubhra made their first contribution in https://github.com/vllm-project/vllm/pull/36545 * @simone-dotolo made their first contribution in https://github.com/vllm-project/vllm/pull/36000 * @sladyn98 made their first contribution in https://github.com/vllm-project/vllm/pull/33503 * @slin1237 made their first contribution in https://github.com/vllm-project/vllm/pull/36938 * @SoluMilken made their first contribution in https://github.com/vllm-project/vllm/pull/36511 * @Srinivasoo7 made their first contribution in https://github.com/vllm-project/vllm/pull/35342 * @stecasta made their first contribution in https://github.com/vllm-project/vllm/pull/35871 * @sungsooha made their first contribution in https://github.com/vllm-project/vllm/pull/34883 * @SunMarc made their first contribution in https://github.com/vllm-project/vllm/pull/36896 * @TQCB made their first contribution in https://github.com/vllm-project/vllm/pull/36165 * @tunglinwood made their first contribution in https://github.com/vllm-project/vllm/pull/36127 * @tusharshetty61 made their first contribution in https://github.com/vllm-project/vllm/pull/36243 * @typer-J made their first contribution in https://github.com/vllm-project/vllm/pull/36578 * @weiguangli-io made their first contribution in https://github.com/vllm-project/vllm/pull/35815 * @wuxun-zhang made their first contribution in https://github.com/vllm-project/vllm/pull/33230 * @XingLiu1 made their first contribution in https://github.com/vllm-project/vllm/pull/35197 * @yanhong-lbh made their first contribution in https://github.com/vllm-project/vllm/pull/32550 * @yitingw1 made their first contribution in https://github.com/vllm-project/vllm/pull/36612 * @yuanheng-zhao made their first contribution in https://github.com/vllm-project/vllm/pull/36106 * @zihaoanllm made their first contribution in https://github.com/vllm-project/vllm/pull/35973