vLLM v0.2.7 — Release Notes & Engineering Impact

## Major Changes * Up to 70% throughput improvement for distributed inference by removing serialization/deserialization overheads * Fix tensor parallelism support for Mixtral + GPTQ/AWQ ## What's Changed * Minor fix for gpu-memory-utilization description by @SuhongMoon in https://github.com/vllm-project/vllm/pull/2162 * [BugFix] Raise error when max_model_len is larger than KV cache size by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2163 * [BugFix] Fix RoPE kernel on long sequences by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2164 * Add SSL arguments to API servers by @HMellor in https://github.com/vllm-project/vllm/pull/2109 * typo fix by @oushu1zhangxiangxuan1 in https://github.com/vllm-project/vllm/pull/2166 * [ROCm] Fixes for GPTQ on ROCm by @kliuae in https://github.com/vllm-project/vllm/pull/2180 * Update Help Text for --gpu-memory-utilization Argument by @SuhongMoon in https://github.com/vllm-project/vllm/pull/2183 * [Minor] Add warning on CUDA graph memory usage by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2182 * Added DeciLM-7b and DeciLM-7b-instruct by @avideci in https://github.com/vllm-project/vllm/pull/2062 * [BugFix] Fix weight loading for Mixtral with TP by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2208 * Make _prepare_sample non blocking and pin memory of CPU input buffers by @hanzhi713 in https://github.com/vllm-project/vllm/pull/2207 * Remove Sampler copy stream by @Yard1 in https://github.com/vllm-project/vllm/pull/2209 * Fix a broken link by @ronensc in https://github.com/vllm-project/vllm/pull/2222 * Disable Ray usage stats collection by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2206 * [BugFix] Fix recovery logic for sequence group by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2186 * Update installation instructions to include CUDA 11.8 xFormers by @skt7 in https://github.com/vllm-project/vllm/pull/2246 * Add "About" Heading to README.md by @blueceiling in https://github.com/vllm-project/vllm/pull/2260 * [BUGFIX] Do not return ignored sentences twice in async llm engine by @zhuohan123 in https://github.com/vllm-project/vllm/pull/2258 * [BUGFIX] Fix API server test by @zhuohan123 in https://github.com/vllm-project/vllm/pull/2270 * [BUGFIX] Fix the path of test prompts by @zhuohan123 in https://github.com/vllm-project/vllm/pull/2273 * [BUGFIX] Fix communication test by @zhuohan123 in https://github.com/vllm-project/vllm/pull/2285 * Add support GPT-NeoX Models without attention biases by @dalgarak in https://github.com/vllm-project/vllm/pull/2301 * [FIX] Fix kernel bug by @jeejeelee in https://github.com/vllm-project/vllm/pull/1959 * fix typo and remove unused code by @esmeetu in https://github.com/vllm-project/vllm/pull/2305 * Enable CUDA graph for GPTQ & SqueezeLLM by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2318 * Fix Gradio example: remove deprecated parameter `concurrency_count` by @ronensc in https://github.com/vllm-project/vllm/pull/2315 * Use NCCL instead of ray for control-plane communication to remove serialization overhead by @zhuohan123 in https://github.com/vllm-project/vllm/pull/2221 * Remove unused const TIMEOUT_TO_PREVENT_DEADLOCK by @ronensc in https://github.com/vllm-project/vllm/pull/2321 * [Minor] Revert the changes in test_cache by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2335 * Bump up to v0.2.7 by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2337 ## New Contributors * @SuhongMoon made their first contribution in https://github.com/vllm-project/vllm/pull/2162 * @HMellor made their first contribution in https://github.com/vllm-project/vllm/pull/2109 * @oushu1zhangxiangxuan1 made their first contribution in https://github.com/vllm-project/vllm/pull/2166 * @kliuae made their first contribution in https://github.com/vllm-project/vllm/pull/2180 * @avideci made their first contribution in https://github.com/vllm-project/vllm/pull/2062 * @hanzhi713 made their first contribution in https://github.com/vllm-project/vllm/pull/2207 * @ronensc made their first contribution in https://github.com/vllm-project/vllm/pull/2222 * @skt7 made their first contribution in https://github.com/vllm-project/vllm/pull/2246 * @blueceiling made their first contribution in https://github.com/vllm-project/vllm/pull/2260 * @dalgarak made their first contribution in https://github.com/vllm-project/vllm/pull/2301 **Full Changelog**: https://github.com/vllm-project/vllm/compare/v0.2.6...v0.2.7