[v0.2.2] Release Tracker

[WoosukKwon]

ETA: Nov 3rd (Fri) - Nov 6th (Mon). Nov 17th (Fri) - 19th (Sun).

Major changes

• Extensive refactoring for better tensor parallelism & quantization support
• Changes in scheduler: from 1D flattened input tensor to 2D tensor
• Bump up to PyTorch v2.1 + CUDA 12.1
• New models: Yi, ChatGLM, Phi
• Added LogitsProcessor API
• Preliminary support for SqueezeLLM

PRs to be merged before the release

• CUDA 12 Upgrade to CUDA 12 #1527
• Yarn YaRN support implementation #1264, YaRN tests #1161
• Phi model [models] Microsoft Phi 1.5 #1664
• [ ] Support embedding inputs Support generation from input embedding #1265

[WoosukKwon]

@zhuohan123 @simon-mo Please feel free to add if you have any PR that should be merged for the next release.

[simon-mo]

I can release the corresponding docker image as well! Hopefully we can get logits processor in as well?

[esmeetu]

It seems that there is something wrong with batch processing at the current main branch.
When i open a api server that serving https://huggingface.co/WizardLM/WizardCoder-1B-V1.0 model, and testing humaneval by using batch requests(164 concurrent reqs) and greedy sampling, it gives me less than 10% on main branch whereas 0.2.1-post1 gives me 23.17%.
related issue: #1570
I have no idea about this and hope this will be addressed at the coming v0.2.2. Thanks!

[miko7879]

Will v0.2.2 work with CUDA 11.8?

[esmeetu]

It seems that there is something wrong with batch processing at the current main branch.
When i open a api server that serving https://huggingface.co/WizardLM/WizardCoder-1B-V1.0 model, and testing humaneval by using batch requests(164 concurrent reqs) and greedy sampling, it gives me less than 10% on main branch whereas 0.2.1-post1 gives me 23.17%.
related issue: #1570
I have no idea about this and hope this will be addressed at the coming v0.2.2. Thanks!

#1546 Fix this.

[shuaiwang2022]

When is the v0.2.2 version scheduled to be released?

[WoosukKwon]

@IncrementalLearning It's in the process. We are planning to release it asap. ETA is Nov 17th (Fri) - 19th (Sun).