vLLM v0.3.3 — Release Notes & Engineering Impact

## Major changes * StarCoder2 support * Performance optimization and LoRA support for Gemma * 2/3/8-bit GPTQ support * Integrate Marlin Kernels for Int4 GPTQ inference * Performance optimization for MoE kernel * [Experimental] AWS Inferentia2 support * [Experimental] Structured output (JSON, Regex) in OpenAI Server ## What's Changed * Update a comment in `benchmark_serving.py` by @ronensc in https://github.com/vllm-project/vllm/pull/2934 * Added early stopping to completion APIs by @Maxusmusti in https://github.com/vllm-project/vllm/pull/2939 * Migrate MistralForCausalLM to LlamaForCausalLM by @esmeetu in https://github.com/vllm-project/vllm/pull/2868 * Use Llama RMSNorm for Gemma by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2974 * chore(vllm): codespell for spell checking by @mspronesti in https://github.com/vllm-project/vllm/pull/2820 * Optimize GeGLU layer in Gemma by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2975 * [FIX] Fix issue #2904 by @44670 in https://github.com/vllm-project/vllm/pull/2983 * Remove Flash Attention in test env by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2982 * Include tokens from prompt phase in `counter_generation_tokens` by @ronensc in https://github.com/vllm-project/vllm/pull/2802 * Fix nvcc not found in vllm-openai image by @zhaoyang-star in https://github.com/vllm-project/vllm/pull/2781 * [Fix] Fix assertion on Mistral YaRN model len by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2984 * Port metrics from `aioprometheus` to `prometheus_client` by @hmellor in https://github.com/vllm-project/vllm/pull/2730 * Add LogProbs for Chat Completions in OpenAI by @jlcmoore in https://github.com/vllm-project/vllm/pull/2918 * Optimized fused MoE Kernel, take 2 by @pcmoritz in https://github.com/vllm-project/vllm/pull/2979 * [Minor] Remove gather_cached_kv kernel by @WoosukKwon in https://github.com/vllm-project/vllm/pull/3043 * [Minor] Remove unused config file by @esmeetu in https://github.com/vllm-project/vllm/pull/3039 * Fix using CuPy for eager mode by @esmeetu in https://github.com/vllm-project/vllm/pull/3037 * Fix stablelm by @esmeetu in https://github.com/vllm-project/vllm/pull/3038 * Support Orion model by @dachengai in https://github.com/vllm-project/vllm/pull/2539 * fix `get_ip` error in pure ipv6 environment by @Jingru in https://github.com/vllm-project/vllm/pull/2931 * [Minor] Fix type annotation in fused moe by @WoosukKwon in https://github.com/vllm-project/vllm/pull/3045 * Support logit bias for OpenAI API by @dylanwhawk in https://github.com/vllm-project/vllm/pull/3027 * [Minor] Fix StableLMEpochForCausalLM -> StableLmForCausalLM by @WoosukKwon in https://github.com/vllm-project/vllm/pull/3046 * Enables GQA support in the prefix prefill kernels by @sighingnow in https://github.com/vllm-project/vllm/pull/3007 * multi-lora documentation fix by @ElefHead in https://github.com/vllm-project/vllm/pull/3064 * Restrict prometheus_client >= 0.18.0 to prevent errors when importing pkgs by @AllenDou in https://github.com/vllm-project/vllm/pull/3070 * Support inference with transformers-neuronx by @liangfu in https://github.com/vllm-project/vllm/pull/2569 * Add LoRA support for Gemma by @WoosukKwon in https://github.com/vllm-project/vllm/pull/3050 * Add Support for 2/3/8-bit GPTQ Quantization Models by @chu-tianxiang in https://github.com/vllm-project/vllm/pull/2330 * Fix: `AttributeError` in OpenAI-compatible server by @jaywonchung in https://github.com/vllm-project/vllm/pull/3018 * add cache_config's info to prometheus metrics. by @AllenDou in https://github.com/vllm-project/vllm/pull/3100 * Support starcoder2 architecture by @sh0416 in https://github.com/vllm-project/vllm/pull/3089 * Fix building from source on WSL by @aliencaocao in https://github.com/vllm-project/vllm/pull/3112 * [Fix] Don't deep-copy LogitsProcessors when copying SamplingParams by @njhill in https://github.com/vllm-project/vllm/pull/3099 * Add guided decoding for OpenAI API server by @felixzhu555 in https://github.com/vllm-project/vllm/pull/2819 * Fix: Output text is always truncated in some models by @HyperdriveHustle in https://github.com/vllm-project/vllm/pull/3016 * Remove exclude_unset in streaming response by @sh0416 in https://github.com/vllm-project/vllm/pull/3143 * docs: Add tutorial on deploying vLLM model with KServe by @terrytangyuan in https://github.com/vllm-project/vllm/pull/2586 * fix relative import path of protocol.py by @Huarong in https://github.com/vllm-project/vllm/pull/3134 * Integrate Marlin Kernels for Int4 GPTQ inference by @robertgshaw2-neuralmagic in https://github.com/vllm-project/vllm/pull/2497 * Bump up to v0.3.3 by @WoosukKwon in https://github.com/vllm-project/vllm/pull/3129 ## New Contributors * @Maxusmusti made their first contribution in https://github.com/vllm-project/vllm/pull/2939 * @44670 made their first contribution in https://github.com/vllm-project/vllm/pull/2983 * @jlcmoore made their first contribution in https://github.com/vllm-project/vllm/pull/2918 * @dachengai made their first contribution in https://github.com/vllm-project/vllm/pull/2539 * @dylanwhawk made their first contribution in https://github.com/vllm-project/vllm/pull/3027 * @ElefHead made their first contribution in https://github.com/vllm-project/vllm/pull/3064 * @AllenDou made their first contribution in https://github.com/vllm-project/vllm/pull/3070 * @jaywonchung made their first contribution in https://github.com/vllm-project/vllm/pull/3018 * @sh0416 made their first contribution in https://github.com/vllm-project/vllm/pull/3089 * @aliencaocao made their first contribution in https://github.com/vllm-project/vllm/pull/3112 * @felixzhu555 made their first contribution in https://github.com/vllm-project/vllm/pull/2819 * @HyperdriveHustle made their first contribution in https://github.com/vllm-project/vllm/pull/3016 * @terrytangyuan made their first contribution in https://github.com/vllm-project/vllm/pull/2586 * @Huarong made their first contribution in https://github.com/vllm-project/vllm/pull/3134 **Full Changelog**: https://github.com/vllm-project/vllm/compare/v0.3.2...v0.3.3