vLLM v0.2.2 — Release Notes & Engineering Impact

## Major changes * Bump up to PyTorch v2.1 + CUDA 12.1 ([vLLM+CUDA 11.8 is also provided](https://vllm.readthedocs.io/en/latest/getting_started/installation.html#install-with-pip)) * Extensive refactoring for better tensor parallelism & quantization support * New models: Yi, ChatGLM, Phi * Changes in scheduler: from 1D flattened input tensor to 2D tensor * AWQ support for all models * Added LogitsProcessor API * Preliminary support for SqueezeLLM ## What's Changed * Change scheduler & input tensor shape by @WoosukKwon in https://github.com/vllm-project/vllm/pull/1381 * Add Mistral 7B to `test_models` by @WoosukKwon in https://github.com/vllm-project/vllm/pull/1366 * fix typo by @WrRan in https://github.com/vllm-project/vllm/pull/1383 * Fix TP bug by @WoosukKwon in https://github.com/vllm-project/vllm/pull/1389 * Fix type hints by @lxrite in https://github.com/vllm-project/vllm/pull/1427 * remove useless statements by @WrRan in https://github.com/vllm-project/vllm/pull/1408 * Pin dependency versions by @thiagosalvatore in https://github.com/vllm-project/vllm/pull/1429 * SqueezeLLM Support by @chooper1 in https://github.com/vllm-project/vllm/pull/1326 * aquila model add rope_scaling by @Sanster in https://github.com/vllm-project/vllm/pull/1457 * fix: don't skip first special token. by @gesanqiu in https://github.com/vllm-project/vllm/pull/1497 * Support repetition_penalty by @beginlner in https://github.com/vllm-project/vllm/pull/1424 * Fix bias in InternLM by @WoosukKwon in https://github.com/vllm-project/vllm/pull/1501 * Delay GPU->CPU sync in sampling by @Yard1 in https://github.com/vllm-project/vllm/pull/1337 * Refactor LLMEngine demo script for clarity and modularity by @iongpt in https://github.com/vllm-project/vllm/pull/1413 * Fix logging issues by @Tostino in https://github.com/vllm-project/vllm/pull/1494 * Add py.typed so consumers of vLLM can get type checking by @jroesch in https://github.com/vllm-project/vllm/pull/1509 * vLLM always places spaces between special tokens by @blahblahasdf in https://github.com/vllm-project/vllm/pull/1373 * [Fix] Fix duplicated logging messages by @zhuohan123 in https://github.com/vllm-project/vllm/pull/1524 * Add dockerfile by @skrider in https://github.com/vllm-project/vllm/pull/1350 * Fix integer overflows in attention & cache ops by @WoosukKwon in https://github.com/vllm-project/vllm/pull/1514 * [Small] Formatter only checks lints in changed files by @cadedaniel in https://github.com/vllm-project/vllm/pull/1528 * Add `MptForCausalLM` key in model_loader by @wenfeiy-db in https://github.com/vllm-project/vllm/pull/1526 * [BugFix] Fix a bug when engine_use_ray=True and worker_use_ray=False and TP>1 by @beginlner in https://github.com/vllm-project/vllm/pull/1531 * Adding a health endpoint by @Fluder-Paradyne in https://github.com/vllm-project/vllm/pull/1540 * Remove `MPTConfig` by @WoosukKwon in https://github.com/vllm-project/vllm/pull/1529 * Force paged attention v2 for long contexts by @Yard1 in https://github.com/vllm-project/vllm/pull/1510 * docs: add description by @lots-o in https://github.com/vllm-project/vllm/pull/1553 * Added logits processor API to sampling params by @noamgat in https://github.com/vllm-project/vllm/pull/1469 * YaRN support implementation by @Yard1 in https://github.com/vllm-project/vllm/pull/1264 * Add Quantization and AutoAWQ to docs by @casper-hansen in https://github.com/vllm-project/vllm/pull/1235 * Support Yi model by @esmeetu in https://github.com/vllm-project/vllm/pull/1567 * ChatGLM2 Support by @GoHomeToMacDonal in https://github.com/vllm-project/vllm/pull/1261 * Upgrade to CUDA 12 by @zhuohan123 in https://github.com/vllm-project/vllm/pull/1527 * [Worker] Fix input_metadata.selected_token_indices in worker by @ymwangg in https://github.com/vllm-project/vllm/pull/1546 * Build CUDA11.8 wheels for release by @WoosukKwon in https://github.com/vllm-project/vllm/pull/1596 * Add Yi model to quantization support by @forpanyang in https://github.com/vllm-project/vllm/pull/1600 * Dockerfile: Upgrade Cuda to 12.1 by @GhaziSyed in https://github.com/vllm-project/vllm/pull/1609 * config parser: add ChatGLM2 seq_length to `_get_and_verify_max_len` by @irasin in https://github.com/vllm-project/vllm/pull/1617 * Fix cpu heavy code in async function _AsyncLLMEngine._run_workers_async by @dominik-schwabe in https://github.com/vllm-project/vllm/pull/1628 * Fix #1474 - gptj AssertionError : assert param_slice.shape == loaded_weight.shape by @lihuahua123 in https://github.com/vllm-project/vllm/pull/1631 * [Minor] Move RoPE selection logic to `get_rope` by @WoosukKwon in https://github.com/vllm-project/vllm/pull/1633 * Add DeepSpeed MII backend to benchmark script by @WoosukKwon in https://github.com/vllm-project/vllm/pull/1649 * TP/quantization/weight loading refactor part 2 - Refactor quantized linear logic and extend quantization support to all models by @zhuohan123 in https://github.com/vllm-project/vllm/pull/1622 * Remove `MptConfig` by @megha95 in https://github.com/vllm-project/vllm/pull/1668 * feat(config): support parsing torch.dtype by @aarnphm in https://github.com/vllm-project/vllm/pull/1641 * Fix loading error when safetensors contains empty tensor by @twaka in https://github.com/vllm-project/vllm/pull/1687 * [Minor] Fix duplication of ignored seq group in engine step by @simon-mo in https://github.com/vllm-project/vllm/pull/1666 * [models] Microsoft Phi 1.5 by @maximzubkov in https://github.com/vllm-project/vllm/pull/1664 * [Fix] Update Supported Models List by @zhuohan123 in https://github.com/vllm-project/vllm/pull/1690 * Return usage for openai requests by @ichernev in https://github.com/vllm-project/vllm/pull/1663 * [Fix] Fix comm test by @zhuohan123 in https://github.com/vllm-project/vllm/pull/1691 * Update the adding-model doc according to the new refactor by @zhuohan123 in https://github.com/vllm-project/vllm/pull/1692 * Add 'not' to this annotation: "#FIXME(woosuk): Do not use internal method" by @linotfan in https://github.com/vllm-project/vllm/pull/1704 * Support Min P Sampler by @esmeetu in https://github.com/vllm-project/vllm/pull/1642 * Read quantization_config in hf config by @WoosukKwon in https://github.com/vllm-project/vllm/pull/1695 * Support download models from www.modelscope.cn by @liuyhwangyh in https://github.com/vllm-project/vllm/pull/1588 * follow up of #1687 when safetensors model contains 0-rank tensors by @twaka in https://github.com/vllm-project/vllm/pull/1696 * Add AWQ support for all models by @WoosukKwon in https://github.com/vllm-project/vllm/pull/1714 * Support fused add rmsnorm for LLaMA by @beginlner in https://github.com/vllm-project/vllm/pull/1667 * [Fix] Fix warning msg on quantization by @WoosukKwon in https://github.com/vllm-project/vllm/pull/1715 * Bump up the version to v0.2.2 by @WoosukKwon in https://github.com/vllm-project/vllm/pull/1689 ## New Contributors * @lxrite made their first contribution in https://github.com/vllm-project/vllm/pull/1427 * @thiagosalvatore made their first contribution in https://github.com/vllm-project/vllm/pull/1429 * @chooper1 made their first contribution in https://github.com/vllm-project/vllm/pull/1326 * @beginlner made their first contribution in https://github.com/vllm-project/vllm/pull/1424 * @iongpt made their first contribution in https://github.com/vllm-project/vllm/pull/1413 * @Tostino made their first contribution in https://github.com/vllm-project/vllm/pull/1494 * @jroesch made their first contribution in https://github.com/vllm-project/vllm/pull/1509 * @skrider made their first contribution in https://github.com/vllm-project/vllm/pull/1350 * @cadedaniel made their first contribution in https://github.com/vllm-project/vllm/pull/1528 * @wenfeiy-db made their first contribution in https://github.com/vllm-project/vllm/pull/1526 * @Fluder-Paradyne made their first contribution in https://github.com/vllm-project/vllm/pull/1540 * @lots-o made their first contribution in https://github.com/vllm-project/vllm/pull/1553 * @noamgat made their first contribution in https://github.com/vllm-project/vllm/pull/1469 * @casper-hansen made their first contribution in https://github.com/vllm-project/vllm/pull/1235 * @GoHomeToMacDonal made their first contribution in https://github.com/vllm-project/vllm/pull/1261 * @ymwangg made their first contribution in https://github.com/vllm-project/vllm/pull/1546 * @forpanyang made their first contribution in https://github.com/vllm-project/vllm/pull/1600 * @GhaziSyed made their first contribution in https://github.com/vllm-project/vllm/pull/1609 * @irasin made their first contribution in https://github.com/vllm-project/vllm/pull/1617 * @dominik-schwabe made their first contribution in https://github.com/vllm-project/vllm/pull/1628 * @lihuahua123 made their first contribution in https://github.com/vllm-project/vllm/pull/1631 * @megha95 made their first contribution in https://github.com/vllm-project/vllm/pull/1668 * @aarnphm made their first contribution in https://github.com/vllm-project/vllm/pull/1641 * @simon-mo made their first contribution in https://github.com/vllm-project/vllm/pull/1666 * @maximzubkov made their first contribution in https://github.com/vllm-project/vllm/pull/1664 * @ichernev made their first contribution in https://github.com/vllm-project/vllm/pull/1663 * @linotfan made their first contribution in https://github.com/vllm-project/vllm/pull/1704 * @liuyhwangyh made their first contribution in https://github.com/vllm-project/vllm/pull/1588 **Full Changelog**: https://github.com/vllm-project/vllm/compare/v0.2.1...v0.2.2