vLLM v0.5.0 — Release Notes & Engineering Impact

# Highlights ## Production Features * [FP8 support](https://docs.vllm.ai/en/stable/quantization/fp8.html) is ready for testing. By quantizing the portion model weights to 8 bit precision float point, the inference speed gets 1.5x boost. Please try it out and let us know your thoughts! (#5352, #5388, #5159, #5238, #5294, #5183, #5144, #5231) * Add OpenAI [Vision API](https://docs.vllm.ai/en/stable/models/vlm.html) support. Currently only LLaVA and LLaVA-NeXT are supported. We are working on adding more models in the next release. (#5237, #5383, #4199, #5374, #4197) * [Speculative Decoding](https://docs.vllm.ai/en/stable/models/spec_decode.html) and [Automatic Prefix Caching](https://docs.vllm.ai/en/stable/automatic_prefix_caching/apc.html) is also ready for testing, we plan to turn them on by default in upcoming releases. (#5400, #5157, #5137, #5324) * Default to multiprocessing backend for single-node distributed case (#5230) * Support bitsandbytes quantization and QLoRA (#4776) ## Hardware Support * Improvements to the Intel CPU CI (#4113, #5241) * Use TORCH_LIBRARY instead of PYBIND11_MODULE for custom ops (#5047) ## Others * [Debugging tips](https://docs.vllm.ai/en/stable/getting_started/debugging.html) documentation (#5409, #5430) * Dynamic Per-Token Activation Quantization (#5037) * Customizable RoPE theta (#5197) * Enable passing multiple LoRA adapters at once to generate() (#5300) * OpenAI `tools` support named functions (#5032) * Support `stream_options` for OpenAI protocol (#5319, #5135) * Update Outlines Integration from `FSM` to `Guide` (#4109) ## What's Changed * [CI/Build] CMakeLists: build all extensions' cmake targets at the same time by @dtrifiro in https://github.com/vllm-project/vllm/pull/5034 * [Kernel] Refactor CUTLASS kernels to always take scales that reside on the GPU by @tlrmchlsmth in https://github.com/vllm-project/vllm/pull/5137 * [Kernel] Update Cutlass fp8 configs by @varun-sundar-rabindranath in https://github.com/vllm-project/vllm/pull/5144 * [Minor] Fix the path typo in loader.py: save_sharded_states.py -> save_sharded_state.py by @dashanji in https://github.com/vllm-project/vllm/pull/5151 * [Bugfix] Fix call to init_logger in openai server by @NadavShmayo in https://github.com/vllm-project/vllm/pull/4765 * [Feature][Kernel] Support bitsandbytes quantization and QLoRA by @chenqianfzh in https://github.com/vllm-project/vllm/pull/4776 * [Bugfix] Remove deprecated @abstractproperty by @zhuohan123 in https://github.com/vllm-project/vllm/pull/5174 * [Bugfix]: Fix issues related to prefix caching example (#5177) by @Delviet in https://github.com/vllm-project/vllm/pull/5180 * [BugFix] Prevent `LLM.encode` for non-generation Models by @robertgshaw2-neuralmagic in https://github.com/vllm-project/vllm/pull/5184 * Update test_ignore_eos by @simon-mo in https://github.com/vllm-project/vllm/pull/4898 * [Frontend][OpenAI] Support for returning max_model_len on /v1/models response by @Avinash-Raj in https://github.com/vllm-project/vllm/pull/4643 * [Kernel][ROCm][AMD] enable fused topk_softmax kernel for moe layer by @divakar-amd in https://github.com/vllm-project/vllm/pull/4927 * [Misc] Simplify code and fix type annotations in `conftest.py` by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/5118 * [Core] Support image processor by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/4197 * [Core] Remove unnecessary copies in flash attn backend by @Yard1 in https://github.com/vllm-project/vllm/pull/5138 * [Kernel] Pass a device pointer into the quantize kernel for the scales by @tlrmchlsmth in https://github.com/vllm-project/vllm/pull/5159 * [CI/BUILD] enable intel queue for longer CPU tests by @zhouyuan in https://github.com/vllm-project/vllm/pull/4113 * [Misc]: Implement CPU/GPU swapping in BlockManagerV2 by @Kaiyang-Chen in https://github.com/vllm-project/vllm/pull/3834 * New CI template on AWS stack by @khluu in https://github.com/vllm-project/vllm/pull/5110 * [FRONTEND] OpenAI `tools` support named functions by @br3no in https://github.com/vllm-project/vllm/pull/5032 * [Bugfix] Support `prompt_logprobs==0` by @toslunar in https://github.com/vllm-project/vllm/pull/5217 * [Bugfix] Add warmup for prefix caching example by @zhuohan123 in https://github.com/vllm-project/vllm/pull/5235 * [Kernel] Enhance MoE benchmarking & tuning script by @WoosukKwon in https://github.com/vllm-project/vllm/pull/4921 * [Bugfix]: During testing, use pytest monkeypatch for safely overriding the env var that indicates the vLLM backend by @afeldman-nm in https://github.com/vllm-project/vllm/pull/5210 * [Bugfix] Fix torch.compile() error when using MultiprocessingGPUExecutor by @zifeitong in https://github.com/vllm-project/vllm/pull/5229 * [CI/Build] Add inputs tests by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/5215 * [Bugfix] Fix a bug caused by pip install setuptools>=49.4.0 for CPU backend by @DamonFool in https://github.com/vllm-project/vllm/pull/5249 * [Kernel] Add back batch size 1536 and 3072 to MoE tuning by @WoosukKwon in https://github.com/vllm-project/vllm/pull/5242 * [CI/Build] Simplify model loading for `HfRunner` by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/5251 * [CI/Build] Reducing CPU CI execution time by @bigPYJ1151 in https://github.com/vllm-project/vllm/pull/5241 * [CI] mark AMD test as softfail to prevent blockage by @simon-mo in https://github.com/vllm-project/vllm/pull/5256 * [Misc] Add transformers version to collect_env.py by @mgoin in https://github.com/vllm-project/vllm/pull/5259 * [Misc] update collect env by @youkaichao in https://github.com/vllm-project/vllm/pull/5261 * [Bugfix] Fix prompt_logprobs when SamplingParams.detokenize is set to True by @zifeitong in https://github.com/vllm-project/vllm/pull/5226 * [Misc] Add CustomOp interface for device portability by @WoosukKwon in https://github.com/vllm-project/vllm/pull/5255 * [Misc] Fix docstring of get_attn_backend by @WoosukKwon in https://github.com/vllm-project/vllm/pull/5271 * [Frontend] OpenAI API server: Add `add_special_tokens` to ChatCompletionRequest (default False) by @tomeras91 in https://github.com/vllm-project/vllm/pull/5278 * [CI] Add nightly benchmarks by @simon-mo in https://github.com/vllm-project/vllm/pull/5260 * [misc] benchmark_serving.py -- add ITL results and tweak TPOT results by @tlrmchlsmth in https://github.com/vllm-project/vllm/pull/5263 * [Kernel] Add GPU architecture guards to the CUTLASS w8a8 kernels to reduce binary size by @tlrmchlsmth in https://github.com/vllm-project/vllm/pull/5157 * [Model] Correct Mixtral FP8 checkpoint loading by @comaniac in https://github.com/vllm-project/vllm/pull/5231 * [BugFix] Apply get_cached_tokenizer to the tokenizer setter of LLM by @DriverSong in https://github.com/vllm-project/vllm/pull/5207 * [Kernel] Re-tune Mixtral MoE configurations for FP8 on H100 by @pcmoritz in https://github.com/vllm-project/vllm/pull/5238 * [Docs] Add Sequoia as sponsors by @simon-mo in https://github.com/vllm-project/vllm/pull/5287 * [Speculative Decoding] Add `ProposerWorkerBase` abstract class by @njhill in https://github.com/vllm-project/vllm/pull/5252 * [BugFix] Fix log message about default max model length by @njhill in https://github.com/vllm-project/vllm/pull/5284 * [Bugfix] Make EngineArgs use named arguments for config construction by @mgoin in https://github.com/vllm-project/vllm/pull/5285 * [Bugfix][Frontend/Core] Don't log exception when AsyncLLMEngine gracefully shuts down. by @wuisawesome in https://github.com/vllm-project/vllm/pull/5290 * [Misc] Skip for logits_scale == 1.0 by @WoosukKwon in https://github.com/vllm-project/vllm/pull/5291 * [Docs] Add Ray Summit CFP by @simon-mo in https://github.com/vllm-project/vllm/pull/5295 * [CI] Disable flash_attn backend for spec decode by @simon-mo in https://github.com/vllm-project/vllm/pull/5286 * [Frontend][Core] Update Outlines Integration from `FSM` to `Guide` by @br3no in https://github.com/vllm-project/vllm/pull/4109 * [CI/Build] Update vision tests by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/5307 * Bugfix: fix broken of download models from modelscope by @liuyhwangyh in https://github.com/vllm-project/vllm/pull/5233 * [Kernel] Retune Mixtral 8x22b configs for FP8 on H100 by @pcmoritz in https://github.com/vllm-project/vllm/pull/5294 * [Frontend] enable passing multiple LoRA adapters at once to generate() by @mgoldey in https://github.com/vllm-project/vllm/pull/5300 * [Core] Avoid copying prompt/output tokens if no penalties are used by @Yard1 in https://github.com/vllm-project/vllm/pull/5289 * [Core] Change LoRA embedding sharding to support loading methods by @Yard1 in https://github.com/vllm-project/vllm/pull/5038 * [Misc] Missing error message for custom ops import by @DamonFool in https://github.com/vllm-project/vllm/pull/5282 * [Feature][Frontend]: Add support for `stream_options` in `ChatCompletionRequest` by @Etelis in https://github.com/vllm-project/vllm/pull/5135 * [Misc][Utils] allow get_open_port to be called for multiple times by @youkaichao in https://github.com/vllm-project/vllm/pull/5333 * [Kernel] Switch fp8 layers to use the CUTLASS kernels by @tlrmchlsmth in https://github.com/vllm-project/vllm/pull/5183 * Remove Ray health check by @Yard1 in https://github.com/vllm-project/vllm/pull/4693 * Addition of lacked ignored_seq_groups in _schedule_chunked_prefill by @JamesLim-sy in https://github.com/vllm-project/vllm/pull/5296 * [Kernel] Dynamic Per-Token Activation Quantization by @dsikka in https://github.com/vllm-project/vllm/pull/5037 * [Frontend] Add OpenAI Vision API Support by @ywang96 in https://github.com/vllm-project/vllm/pull/5237 * [Misc] Remove unused cuda_utils.h in CPU backend by @DamonFool in https://github.com/vllm-project/vllm/pull/5345 * fix DbrxFusedNormAttention missing cache_config by @Calvinnncy97 in https://github.com/vllm-project/vllm/pull/5340 * [Bug Fix] Fix the support check for FP8 CUTLASS by @cli99 in https://github.com/vllm-project/vllm/pull/5352 * [Misc] Add args for selecting distributed executor to benchmarks by @BKitor in https://github.com/vllm-project/vllm/pull/5335 * [ROCm][AMD] Use pytorch sdpa math backend to do naive attention by @hongxiayang in https://github.com/vllm-project/vllm/pull/4965 * [CI/Test] improve robustness of test by replacing del with context manager (hf_runner) by @youkaichao in https://github.com/vllm-project/vllm/pull/5347 * [CI/Test] improve robustness of test by replacing del with context manager (vllm_runner) by @youkaichao in https://github.com/vllm-project/vllm/pull/5357 * [Misc][Breaking] Change FP8 checkpoint format from act_scale -> input_scale by @mgoin in https://github.com/vllm-project/vllm/pull/5353 * [Core][CUDA Graph] add output buffer for cudagraph to reduce memory footprint by @youkaichao in https://github.com/vllm-project/vllm/pull/5074 * [mis][ci/test] fix flaky test in tests/test_sharded_state_loader.py by @youkaichao in https://github.com/vllm-project/vllm/pull/5361 * [Kernel][Misc] Use TORCH_LIBRARY instead of PYBIND11_MODULE for custom ops by @bnellnm in https://github.com/vllm-project/vllm/pull/5047 * [Bugfix] Fix KeyError: 1 When Using LoRA adapters by @BlackBird-Coding in https://github.com/vllm-project/vllm/pull/5164 * [Misc] Update to comply with the new `compressed-tensors` config by @dsikka in https://github.com/vllm-project/vllm/pull/5350 * [Frontend][Misc] Enforce Pixel Values as Input Type for VLMs in API Server by @ywang96 in https://github.com/vllm-project/vllm/pull/5374 * [misc][typo] fix typo by @youkaichao in https://github.com/vllm-project/vllm/pull/5372 * [Misc] Improve error message when LoRA parsing fails by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/5194 * [Model] Initial support for LLaVA-NeXT by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/4199 * [Feature][Frontend]: Continued `stream_options` implementation also in CompletionRequest by @Etelis in https://github.com/vllm-project/vllm/pull/5319 * [Bugfix] Fix LLaVA-NeXT by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/5380 * [ci] Use small_cpu_queue for doc build by @khluu in https://github.com/vllm-project/vllm/pull/5331 * [ci] Mount buildkite agent on Docker container to upload benchmark results by @khluu in https://github.com/vllm-project/vllm/pull/5330 * [Docs] Add Docs on Limitations of VLM Support by @ywang96 in https://github.com/vllm-project/vllm/pull/5383 * [Docs] Alphabetically sort sponsors by @WoosukKwon in https://github.com/vllm-project/vllm/pull/5386 * Bump version to v0.5.0 by @simon-mo in https://github.com/vllm-project/vllm/pull/5384 * [Doc] Add documentation for FP8 W8A8 by @mgoin in https://github.com/vllm-project/vllm/pull/5388 * [ci] Fix Buildkite agent path by @khluu in https://github.com/vllm-project/vllm/pull/5392 * [Misc] Various simplifications and typing fixes by @njhill in https://github.com/vllm-project/vllm/pull/5368 * [Bugfix] OpenAI entrypoint limits logprobs while ignoring server defined --max-logprobs by @maor-ps in https://github.com/vllm-project/vllm/pull/5312 * [Bugfix][Frontend] Cleanup "fix chat logprobs" by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/5026 * [Doc] add debugging tips by @youkaichao in https://github.com/vllm-project/vllm/pull/5409 * [Doc][Typo] Fixing Missing Comma by @ywang96 in https://github.com/vllm-project/vllm/pull/5403 * [Misc] Remove VLLM_BUILD_WITH_NEURON env variable by @WoosukKwon in https://github.com/vllm-project/vllm/pull/5389 * [CI] docfix by @rkooo567 in https://github.com/vllm-project/vllm/pull/5410 * [Speculative decoding] Initial spec decode docs by @cadedaniel in https://github.com/vllm-project/vllm/pull/5400 * [Doc] Add an automatic prefix caching section in vllm documentation by @KuntaiDu in https://github.com/vllm-project/vllm/pull/5324 * [Docs] [Spec decode] Fix docs error in code example by @cadedaniel in https://github.com/vllm-project/vllm/pull/5427 * [Bugfix] Fix `MultiprocessingGPUExecutor.check_health` when world_size == 1 by @jsato8094 in https://github.com/vllm-project/vllm/pull/5254 * [Bugfix] fix lora_dtype value type in arg_utils.py by @c3-ali in https://github.com/vllm-project/vllm/pull/5398 * [Frontend] Customizable RoPE theta by @sasha0552 in https://github.com/vllm-project/vllm/pull/5197 * [Core][Distributed] add same-node detection by @youkaichao in https://github.com/vllm-project/vllm/pull/5369 * [Core][Doc] Default to multiprocessing for single-node distributed case by @njhill in https://github.com/vllm-project/vllm/pull/5230 * [Doc] add common case for long waiting time by @youkaichao in https://github.com/vllm-project/vllm/pull/5430 ## New Contributors * @dtrifiro made their first contribution in https://github.com/vllm-project/vllm/pull/5034 * @varun-sundar-rabindranath made their first contribution in https://github.com/vllm-project/vllm/pull/5144 * @dashanji made their first contribution in https://github.com/vllm-project/vllm/pull/5151 * @chenqianfzh made their first contribution in https://github.com/vllm-project/vllm/pull/4776 * @Delviet made their first contribution in https://github.com/vllm-project/vllm/pull/5180 * @Avinash-Raj made their first contribution in https://github.com/vllm-project/vllm/pull/4643 * @zhouyuan made their first contribution in https://github.com/vllm-project/vllm/pull/4113 * @Kaiyang-Chen made their first contribution in https://github.com/vllm-project/vllm/pull/3834 * @khluu made their first contribution in https://github.com/vllm-project/vllm/pull/5110 * @toslunar made their first contribution in https://github.com/vllm-project/vllm/pull/5217 * @DamonFool made their first contribution in https://github.com/vllm-project/vllm/pull/5249 * @tomeras91 made their first contribution in https://github.com/vllm-project/vllm/pull/5278 * @DriverSong made their first contribution in https://github.com/vllm-project/vllm/pull/5207 * @mgoldey made their first contribution in https://github.com/vllm-project/vllm/pull/5300 * @JamesLim-sy made their first contribution in https://github.com/vllm-project/vllm/pull/5296 * @Calvinnncy97 made their first contribution in https://github.com/vllm-project/vllm/pull/5340 * @cli99 made their first contribution in https://github.com/vllm-project/vllm/pull/5352 * @BKitor made their first contribution in https://github.com/vllm-project/vllm/pull/5335 * @BlackBird-Coding made their first contribution in https://github.com/vllm-project/vllm/pull/5164 * @maor-ps made their first contribution in https://github.com/vllm-project/vllm/pull/5312 * @c3-ali made their first contribution in https://github.com/vllm-project/vllm/pull/5398 **Full Changelog**: https://github.com/vllm-project/vllm/compare/v0.4.3...v0.5.0