Upstream merge 24 12 16 #330

gshtras · 2024-12-16T17:43:07Z

Also deprecating sync openai entrypoint

Signed-off-by: Richard Liu <ricliu@google.com> Co-authored-by: Kevin H. Luu <kevin@anyscale.com>

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

Signed-off-by: mgoin <michael@neuralmagic.com> Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>

…er for Mamba (vllm-project#10739) Signed-off-by: Isotr0py <2037008807@qq.com>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>

Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>

…roject#11031) Signed-off-by: youkaichao <youkaichao@gmail.com>

…n PixtralHF. (vllm-project#11043)

Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>

…ect#11039) Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>

Signed-off-by: Flavia Beo <flavia.beo@ibm.com>

Signed-off-by: Russell Bryant <rbryant@redhat.com>

…t#11059) Signed-off-by: youkaichao <youkaichao@gmail.com>

…oject#11061) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>

Signed-off-by: mzusman <mor.zusmann@gmail.com>

…lm-project#11069) Signed-off-by: Max de Bayser <mbayser@br.ibm.com>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…t#11088)

…ject#11076) Signed-off-by: Russell Bryant <rbryant@redhat.com>

…t#11080) Signed-off-by: B-201 <Joy25810@foxmail.com>

…t#11073) Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>

Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>

…oject#11186) Signed-off-by: Cody Yu <hao.yu.cody@gmail.com>

Co-authored-by: dhuangnm <dhuang@MacBook-Pro-2.local>

…#11140) Signed-off-by: Russell Bryant <rbryant@redhat.com>

Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>

…lm-project#11200) Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

…llm-project#11150) Signed-off-by: Brad Hilton <brad.hilton.nw@gmail.com>

…m-project#11199) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Signed-off-by: Mark McLoughlin <markmc@redhat.com>

…applying a priority queue and lazy deletion (vllm-project#7209)

…t#11201)

Signed-off-by: youkaichao <youkaichao@gmail.com>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…ct#11213) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

) Signed-off-by: Kuntai Du <kuntai@uchicago.edu>

…llm-project#10884) Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>

…m-project#11214) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

…st (vllm-project#11219)

…ject#11212) Signed-off-by: drikster80 <ed.sealing@gmail.com> Co-authored-by: drikster80 <ed.sealing@gmail.com>

…#11198) Signed-off-by: Isotr0py <2037008807@qq.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

…t#11228) Signed-off-by: Isotr0py <2037008807@qq.com>

richardsliu and others added 30 commits December 9, 2024 23:07

Build tpu image in release pipeline (vllm-project#10936)

5ed5d5f

Signed-off-by: Richard Liu <ricliu@google.com> Co-authored-by: Kevin H. Luu <kevin@anyscale.com>

[V1] Do not store None in self.generators (vllm-project#11038)

6faec54

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

[Docs] Add dedicated tool calling page to docs (vllm-project#10554)

6d52528

Signed-off-by: mgoin <michael@neuralmagic.com> Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>

[Model] Add has_weight to RMSNorm and re-enable weights loading track…

d1f6d1c

…er for Mamba (vllm-project#10739) Signed-off-by: Isotr0py <2037008807@qq.com>

[Bugfix] Fix usage of deprecated decorator (vllm-project#11025)

391d7b2

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Frontend] Use request id from header (vllm-project#10968)

980ad39

Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>

[Pixtral] Improve loading (vllm-project#11040)

bc192a2

[V1] Multiprocessing Tensor Parallel Support for v1 (vllm-project#9856)

28b3a1c

Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>

monitor metrics of tokens per step using cudagraph batchsizes (vllm-p…

ebf7780

…roject#11031) Signed-off-by: youkaichao <youkaichao@gmail.com>

[Bugfix] Fix xgrammar failing to read a vocab_size from LlavaConfig o…

e35879c

…n PixtralHF. (vllm-project#11043)

Update README.md (vllm-project#11034)

bfd6104

[Bugfix] cuda error running llama 3.2 (vllm-project#11047)

82c73fd

Add example of helm chart for vllm deployment on k8s (vllm-project#9199)

fe2e10c

Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>

[Bugfix] Handle <|tool_call|> token in granite tool parser (vllm-proj…

beb16b2

…ect#11039) Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>

[Misc][LoRA] Add PEFTHelper for LoRA (vllm-project#11003)

d05f886

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

[Bugfix] Backport request id validation to v0 (vllm-project#11036)

9b9cef3

Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>

[BUG] Remove token param vllm-project#10921 (vllm-project#11022)

250ee65

Signed-off-by: Flavia Beo <flavia.beo@ibm.com>

[Core] Update to outlines >= 0.1.8 (vllm-project#10576)

e739194

Signed-off-by: Russell Bryant <rbryant@redhat.com>

[torch.compile] add a flag to track batchsize statistics (vllm-projec…

75f89dc

…t#11059) Signed-off-by: youkaichao <youkaichao@gmail.com>

[V1][Bugfix] Always set enable_chunked_prefill = True for V1 (vllm-pr…

134810b

…oject#11061) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

[Bugfix] Fix Mamba multistep (vllm-project#11071)

9a93973

Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>

[Misc] LoRA + Chunked Prefill (vllm-project#9057)

d5c5154

[Model] PP support for Mamba-like models (vllm-project#10992)

ffa48c9

Signed-off-by: mzusman <mor.zusmann@gmail.com>

Fix streaming for granite tool call when <|tool_call|> is present (vl…

e39400a

…lm-project#11069) Signed-off-by: Max de Bayser <mbayser@br.ibm.com>

[CI/Build] Check transformers v4.47 (vllm-project#10991)

2e33fe4

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[ci/build] Fix AMD CI dependencies (vllm-project#11087)

3fb4b4f

[ci/build] Fix entrypoints test and pin outlines version (vllm-projec…

9974fca

…t#11088)

[Core] v1: Use atexit to handle engine core client shutdown (vllm-pro…

61b1d2f

…ject#11076) Signed-off-by: Russell Bryant <rbryant@redhat.com>

[Bugfix] Fix Idefics3 fails during multi-image inference (vllm-projec…

2e32f5d

…t#11080) Signed-off-by: B-201 <Joy25810@foxmail.com>

[Bugfix]: Clamp -inf logprob values in prompt_logprobs (vllm-projec…

40766ca

…t#11073) Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>

tlrmchlsmth and others added 27 commits December 13, 2024 17:02

[V1][Bugfix] Fix EngineCoreProc profile (vllm-project#11185)

4b5b8a6

Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>

[Bugfix][V1] Re-compute an entire block when fully cache hit (vllm-pr…

9855aea

…oject#11186) Signed-off-by: Cody Yu <hao.yu.cody@gmail.com>

update compressed-tensors to latest version (vllm-project#11183)

24a3d12

Co-authored-by: dhuangnm <dhuang@MacBook-Pro-2.local>

[Core] Update outlines and increase its threadpool size (vllm-project…

4825926

…#11140) Signed-off-by: Russell Bryant <rbryant@redhat.com>

[V1][Bugfix] Fix V1 TP trust-remote-code (vllm-project#11182)

ea7bd68

Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>

[Misc] Minor improvements to the readability of PunicaWrapperBase (vl…

3cb5769

…lm-project#11200) Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

[Frontend] Add logits_processors as an extra completion argument (v…

9c3dadd

…llm-project#11150) Signed-off-by: Brad Hilton <brad.hilton.nw@gmail.com>

[VLM] Fully dynamic prompt replacement in merged input processor (vll…

93abf23

…m-project#11199) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Enable mypy checking on V1 code (vllm-project#11105)

6d917d0

Signed-off-by: Mark McLoughlin <markmc@redhat.com>

[Performance][Core] Optimize the performance of evictor v1 and v2 by …

8869368

…applying a priority queue and lazy deletion (vllm-project#7209)

[[Misc]Upgrade bitsandbytes to the latest version 0.45.0 (vllm-projec…

15859f2

…t#11201)

[torch.compile] allow tracking forward time (vllm-project#11081)

a1c0205

Signed-off-by: youkaichao <youkaichao@gmail.com>

[Misc] Clean up multi-modal processor (vllm-project#11207)

b10609e

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Bugfix] Fix error handling of unsupported sliding window (vllm-proje…

96d673e

…ct#11213) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Doc] add documentation for disaggregated prefilling (vllm-project#11197

38e599d

) Signed-off-by: Kuntai Du <kuntai@uchicago.edu>

[Core] Support disaggregated prefill with Mooncake Transfer Engine (v…

d263bd9

…llm-project#10884) Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>

[V1][Minor] Cache np arange to reduce input preparation overhead (vll…

25ebed2

…m-project#11214) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

Update deploying_with_k8s.rst (vllm-project#10922)

da6f409

[Bugfix] Fix block size validation (vllm-project#10938)

69ba344

[Bugfix] Fix the default value for temperature in ChatCompletionReque…

17138af

…st (vllm-project#11219)

WIP: [CI/Build] simplify Dockerfile build for ARM64 / GH200 (vllm-pro…

b3b1526

…ject#11212) Signed-off-by: drikster80 <ed.sealing@gmail.com> Co-authored-by: drikster80 <ed.sealing@gmail.com>

[Model] Support Cohere2ForCausalLM (Cohere R7B) (vllm-project#11203)

bddbbcb

[Model] Refactor Ultravox to use merged input processor (vllm-project…

d927dbc

…#11198) Signed-off-by: Isotr0py <2037008807@qq.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

[Doc] Reorder vision language examples in alphabet order (vllm-projec…

2ca830d

…t#11228) Signed-off-by: Isotr0py <2037008807@qq.com>

Merge remote-tracking branch 'upstream/main'

1a8e549

Deprecating sync_openai

78440dc

Remove new irrelevant action

ddec133

shajrawi approved these changes Dec 16, 2024

View reviewed changes

gshtras merged commit 1f0f4c6 into main Dec 16, 2024
10 of 11 checks passed

gshtras deleted the upstream_merge_24_12_16 branch December 16, 2024 21:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upstream merge 24 12 16 #330

Upstream merge 24 12 16 #330

gshtras commented Dec 16, 2024

Upstream merge 24 12 16 #330

Upstream merge 24 12 16 #330

Conversation

gshtras commented Dec 16, 2024