vLLM v0.3.1 — Release Notes & Engineering Impact

## Major Changes This version fixes the following major bugs: * Memory leak with distributed execution. (Solved by using CuPY for collective communication). * Support for Python 3.8. Also with many smaller bug fixes listed below. ## What's Changed * Fixes assertion failure in prefix caching: the lora index mapping should respect `prefix_len`. by @sighingnow in https://github.com/vllm-project/vllm/pull/2688 * fix some bugs about parameter description by @zspo in https://github.com/vllm-project/vllm/pull/2689 * [Minor] Fix test_cache.py CI test failure by @pcmoritz in https://github.com/vllm-project/vllm/pull/2684 * Add unit test for Mixtral MoE layer by @pcmoritz in https://github.com/vllm-project/vllm/pull/2677 * Refactor Prometheus and Add Request Level Metrics by @rib-2 in https://github.com/vllm-project/vllm/pull/2316 * Add Internlm2 by @Leymore in https://github.com/vllm-project/vllm/pull/2666 * Fix compile error when using rocm by @zhaoyang-star in https://github.com/vllm-project/vllm/pull/2648 * fix python 3.8 syntax by @simon-mo in https://github.com/vllm-project/vllm/pull/2716 * Update README for meetup slides by @simon-mo in https://github.com/vllm-project/vllm/pull/2718 * Use revision when downloading the quantization config file by @Pernekhan in https://github.com/vllm-project/vllm/pull/2697 * remove hardcoded `device="cuda" ` to support more device by @jikunshang in https://github.com/vllm-project/vllm/pull/2503 * fix length_penalty default value to 1.0 by @zspo in https://github.com/vllm-project/vllm/pull/2667 * Add one example to run batch inference distributed on Ray by @c21 in https://github.com/vllm-project/vllm/pull/2696 * docs: update langchain serving instructions by @mspronesti in https://github.com/vllm-project/vllm/pull/2736 * Set&Get llm internal tokenizer instead of the TokenizerGroup by @dancingpipi in https://github.com/vllm-project/vllm/pull/2741 * Remove eos tokens from output by default by @zcnrex in https://github.com/vllm-project/vllm/pull/2611 * add requirement: triton >= 2.1.0 by @whyiug in https://github.com/vllm-project/vllm/pull/2746 * [Minor] Fix benchmark_latency by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2765 * [ROCm] Fix some kernels failed unit tests by @hongxiayang in https://github.com/vllm-project/vllm/pull/2498 * Set local logging level via env variable by @gardberg in https://github.com/vllm-project/vllm/pull/2774 * [ROCm] Fixup arch checks for ROCM by @dllehr-amd in https://github.com/vllm-project/vllm/pull/2627 * Add fused top-K softmax kernel for MoE by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2769 * fix issue when model parameter is not a model id but path of the model. by @liuyhwangyh in https://github.com/vllm-project/vllm/pull/2489 * [Minor] More fix of test_cache.py CI test failure by @LiuXiaoxuanPKU in https://github.com/vllm-project/vllm/pull/2750 * [ROCm] Fix build problem resulted from previous commit related to FP8 kv-cache support by @hongxiayang in https://github.com/vllm-project/vllm/pull/2790 * Add documentation on how to do incremental builds by @pcmoritz in https://github.com/vllm-project/vllm/pull/2796 * [Ray] Integration compiled DAG off by default by @rkooo567 in https://github.com/vllm-project/vllm/pull/2471 * Disable custom all reduce by default by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2808 * [ROCm] support Radeon™ 7900 series (gfx1100) without using flash-attention by @hongxiayang in https://github.com/vllm-project/vllm/pull/2768 * Add documentation section about LoRA by @pcmoritz in https://github.com/vllm-project/vllm/pull/2834 * Refactor 2 awq gemm kernels into m16nXk32 by @zcnrex in https://github.com/vllm-project/vllm/pull/2723 * Serving Benchmark Refactoring by @ywang96 in https://github.com/vllm-project/vllm/pull/2433 * [CI] Ensure documentation build is checked in CI by @simon-mo in https://github.com/vllm-project/vllm/pull/2842 * Refactor llama family models by @esmeetu in https://github.com/vllm-project/vllm/pull/2637 * Revert "Refactor llama family models" by @pcmoritz in https://github.com/vllm-project/vllm/pull/2851 * Use CuPy for CUDA graphs by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2811 * Remove Yi model definition, please use `LlamaForCausalLM` instead by @pcmoritz in https://github.com/vllm-project/vllm/pull/2854 * Add LoRA support for Mixtral by @tterrysun in https://github.com/vllm-project/vllm/pull/2831 * Migrate InternLMForCausalLM to LlamaForCausalLM by @pcmoritz in https://github.com/vllm-project/vllm/pull/2860 * Fix internlm after https://github.com/vllm-project/vllm/pull/2860 by @pcmoritz in https://github.com/vllm-project/vllm/pull/2861 * [Fix] Fix memory profiling when GPU is used by multiple processes by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2863 * Fix docker python version by @NikolaBorisov in https://github.com/vllm-project/vllm/pull/2845 * Migrate AquilaForCausalLM to LlamaForCausalLM by @esmeetu in https://github.com/vllm-project/vllm/pull/2867 * Don't use cupy NCCL for AMD backends by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2855 * Align LoRA code between Mistral and Mixtral (fixes #2875) by @pcmoritz in https://github.com/vllm-project/vllm/pull/2880 * [BugFix] Fix GC bug for `LLM` class by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2882 * Fix decilm.py by @pcmoritz in https://github.com/vllm-project/vllm/pull/2883 * [ROCm] Dockerfile fix for flash-attention build by @hongxiayang in https://github.com/vllm-project/vllm/pull/2885 * Prefix Caching- fix t4 triton error by @caoshiyi in https://github.com/vllm-project/vllm/pull/2517 * Bump up to v0.3.1 by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2887 ## New Contributors * @sighingnow made their first contribution in https://github.com/vllm-project/vllm/pull/2688 * @rib-2 made their first contribution in https://github.com/vllm-project/vllm/pull/2316 * @Leymore made their first contribution in https://github.com/vllm-project/vllm/pull/2666 * @Pernekhan made their first contribution in https://github.com/vllm-project/vllm/pull/2697 * @jikunshang made their first contribution in https://github.com/vllm-project/vllm/pull/2503 * @c21 made their first contribution in https://github.com/vllm-project/vllm/pull/2696 * @zcnrex made their first contribution in https://github.com/vllm-project/vllm/pull/2611 * @whyiug made their first contribution in https://github.com/vllm-project/vllm/pull/2746 * @gardberg made their first contribution in https://github.com/vllm-project/vllm/pull/2774 * @dllehr-amd made their first contribution in https://github.com/vllm-project/vllm/pull/2627 * @rkooo567 made their first contribution in https://github.com/vllm-project/vllm/pull/2471 * @ywang96 made their first contribution in https://github.com/vllm-project/vllm/pull/2433 * @tterrysun made their first contribution in https://github.com/vllm-project/vllm/pull/2831 **Full Changelog**: https://github.com/vllm-project/vllm/compare/v0.3.0...v0.3.1