vLLM v0.2.4 — Release Notes & Engineering Impact

## Major changes * Mixtral model support (officially from @mistralai) * AMD GPU support (collaboration with @embeddedllm) ## What's Changed * add custom server params by @esmeetu in https://github.com/vllm-project/vllm/pull/1868 * support ChatGLMForConditionalGeneration by @dancingpipi in https://github.com/vllm-project/vllm/pull/1932 * Save pytorch profiler output for latency benchmark by @Yard1 in https://github.com/vllm-project/vllm/pull/1871 * Fix typo in adding_model.rst by @petergtz in https://github.com/vllm-project/vllm/pull/1947 * Make InternLM follow `rope_scaling` in `config.json` by @theFool32 in https://github.com/vllm-project/vllm/pull/1956 * Fix quickstart.rst example by @gottlike in https://github.com/vllm-project/vllm/pull/1964 * Adding number of nvcc_threads during build as envar by @AguirreNicolas in https://github.com/vllm-project/vllm/pull/1893 * fix typo in getenv call by @dskhudia in https://github.com/vllm-project/vllm/pull/1972 * [Continuation] Merge EmbeddedLLM/vllm-rocm into vLLM main by @tjtanaa in https://github.com/vllm-project/vllm/pull/1836 * Fix Baichuan2-7B-Chat by @firebook in https://github.com/vllm-project/vllm/pull/1987 * [Docker] Add cuda arch list as build option by @simon-mo in https://github.com/vllm-project/vllm/pull/1950 * Fix for KeyError on Loading LLaMA by @imgaojun in https://github.com/vllm-project/vllm/pull/1978 * [Minor] Fix code style for baichuan by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2003 * Fix OpenAI server completion_tokens referenced before assignment by @js8544 in https://github.com/vllm-project/vllm/pull/1996 * [Minor] Add comment on skipping rope caches by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2004 * Replace head_mapping params with num_kv_heads to attention kernel. by @wbn03 in https://github.com/vllm-project/vllm/pull/1997 * Fix completion API echo and logprob combo by @simon-mo in https://github.com/vllm-project/vllm/pull/1992 * Mixtral 8x7B support by @pierrestock in https://github.com/vllm-project/vllm/pull/2011 * Minor fixes for Mixtral by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2015 * Change load format for Mixtral by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2028 * Update run_on_sky.rst by @eltociear in https://github.com/vllm-project/vllm/pull/2025 * Update requirements.txt for mixtral by @0-hero in https://github.com/vllm-project/vllm/pull/2029 * Revert #2029 by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2030 * [Minor] Fix latency benchmark script by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2035 * [Minor] Fix type annotation in Mixtral by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2036 * Update README.md to add megablocks requirement for mixtral by @0-hero in https://github.com/vllm-project/vllm/pull/2033 * [Minor] Fix import error msg for megablocks by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2038 * Bump up to v0.2.4 by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2034 ## New Contributors * @dancingpipi made their first contribution in https://github.com/vllm-project/vllm/pull/1932 * @petergtz made their first contribution in https://github.com/vllm-project/vllm/pull/1947 * @theFool32 made their first contribution in https://github.com/vllm-project/vllm/pull/1956 * @gottlike made their first contribution in https://github.com/vllm-project/vllm/pull/1964 * @AguirreNicolas made their first contribution in https://github.com/vllm-project/vllm/pull/1893 * @dskhudia made their first contribution in https://github.com/vllm-project/vllm/pull/1972 * @tjtanaa made their first contribution in https://github.com/vllm-project/vllm/pull/1836 * @firebook made their first contribution in https://github.com/vllm-project/vllm/pull/1987 * @imgaojun made their first contribution in https://github.com/vllm-project/vllm/pull/1978 * @js8544 made their first contribution in https://github.com/vllm-project/vllm/pull/1996 * @wbn03 made their first contribution in https://github.com/vllm-project/vllm/pull/1997 * @pierrestock made their first contribution in https://github.com/vllm-project/vllm/pull/2011 * @0-hero made their first contribution in https://github.com/vllm-project/vllm/pull/2029 **Full Changelog**: https://github.com/vllm-project/vllm/compare/v0.2.3...v0.2.4