Code

fastforward on main#2

Code

Closed

SolitaryThinker wants to merge 138 commits intohao-ai-lab:mainhao-ai-lab/vllm:mainfrom vllm-project:mainvllm-project/vllm:mainCopy head branch name to clipboard

+25,844-6,386

Commits

Commits on Feb 3, 2025

• [Doc] Deprecate Discord (#12668)

  zhuohan123

  authored
  326fcc8

  Copy full SHA for 326fcc8

• [Kernel] port sgl moe_align_block_size kernels (#12574)

  Show description for 95460fc
  chenyang78

  authored
  95460fc

  Copy full SHA for 95460fc

• make sure mistral_common not imported for non-mistral models (#12669)

  Show description for 20579c0
  youkaichao

  authored
  20579c0

  Copy full SHA for 20579c0

• Properly check if all fused layers are in the list of targets (#12666)

  Show description for c5932e5
  eldarkurtic

  authored
  c5932e5

  Copy full SHA for c5932e5

• Fix for attention layers to remain unquantized during moe_wn16 quant (#12570)

  Show description for b998645
  

  31 peopleauthored
  b998645

  Copy full SHA for b998645

• [cuda] manually import the correct pynvml module (#12679)

  Show description for ad4a9dc
  youkaichao

  authored
  ad4a9dc

  Copy full SHA for ad4a9dc

• [ci/build] fix gh200 test (#12681)

  Show description for 1298a40
  youkaichao

  authored
  1298a40

  Copy full SHA for 1298a40

• [Model]: Add transformers backend support (#11330)

  Show description for a1a2aaa
  

  5 peopleauthored
  a1a2aaa

  Copy full SHA for a1a2aaa

• [Misc] Fix improper placement of SPDX header in scripts (#12694)

  Show description for 33e0602
  russellb

  authored
  33e0602

  Copy full SHA for 33e0602

• [Bugfix][Kernel] Fix per-token/per-channel quantization for Hopper scaled mm (#12696)

  Show description for c11de33
  tlrmchlsmth

  authored
  c11de33

  Copy full SHA for c11de33

• Squelch MLA warning for Compressed-Tensors Models (#12704)

  Show description for 6dd5e52
  kylesayrs

  authored
  6dd5e52

  Copy full SHA for 6dd5e52

• [Model] Add Deepseek V3 fp8_w8a8 configs for B200 (#12707)

  kushanam

  authored
  4797dad

  Copy full SHA for 4797dad

• [MISC] Remove model input dumping when exception (#12582)

  Show description for cf58b9c
  comaniac

  authored
  cf58b9c

  Copy full SHA for cf58b9c

• [V1] Revert uncache_blocks and support recaching full blocks (#12415)

  Show description for 5095e96
  comaniac

  authored
  5095e96

  Copy full SHA for 5095e96

Commits on Feb 4, 2025

• [Core] Improve hash collision avoidance in prefix caching (#12621)

  Show description for 73b35cc
  russellb

  authored
  73b35cc

  Copy full SHA for 73b35cc

• Support Pixtral-Large HF by using llava multimodal_projector_bias config (#12710)

  Show description for 5d98d56
  mgoin

  authored
  5d98d56

  Copy full SHA for 5d98d56

• [Doc] Replace ibm-fms with ibm-ai-platform (#12709)

  Show description for bb392af
  tdoublep

  authored
  bb392af

  Copy full SHA for bb392af

• [Quant] Fix use_mla TypeError and support loading pure-sparsity Compressed Tensors configs (#12711)

  kylesayrs

  authored
  4896d0c

  Copy full SHA for 4896d0c

• [AMD][ROCm] Enable DeepSeek model on ROCm (#12662)

  Show description for c36ac98
  

  hongxiayang

  and

  mawong-amd

  authored
  c36ac98

  Copy full SHA for c36ac98

• [Misc] Add BNB quantization for Whisper (#12381)

  Show description for 96b2362
  jeejeelee

  authored
  96b2362

  Copy full SHA for 96b2362

• [VLM] Merged multi-modal processor for InternVL-based models (#12553)

  Show description for d1ca7df
  

  DarkLight1337

  and

  Isotr0py

  authored
  d1ca7df

  Copy full SHA for d1ca7df

• [V1] Remove scheduling constraint on partial requests (#12674)

  Show description for 18a88fc
  WoosukKwon

  authored
  18a88fc

  Copy full SHA for 18a88fc

• [VLM] merged multimodal processor and V1 support for idefics3 (#12660)

  Show description for 815079d
  

  Isotr0py

  and

  DarkLight1337

  authored
  815079d

  Copy full SHA for 815079d

• [Bugfix] Fix loading of fine-tuned models based on Phi-3-Small (#12689)

  Show description for 6469038
  

  mgtk77

  and

  MicGree

  authored
  6469038

  Copy full SHA for 6469038

• Avoid unnecessary multi-modal input data copy when len(batch) == 1 (#12722)

  Show description for 62467a8
  imkero

  authored
  62467a8

  Copy full SHA for 62467a8

• [Build] update requirements of no-device for plugin usage (#12630)

  Show description for 649550f
  sducouedic

  authored
  649550f

  Copy full SHA for 649550f

• [Bugfix] Fix CI failures for InternVL and Mantis models (#12728)

  Show description for 18016a5
  DarkLight1337

  authored
  18016a5

  Copy full SHA for 18016a5

Commits on Feb 5, 2025

• [V1][Metrics] Add request_success_total counter, labelled with finish reason (#12579)

  Show description for 233df6f
  markmc

  authored
  233df6f

  Copy full SHA for 233df6f

• [Perf] Mem align KV caches for CUDA devices (MLA perf improvement) (#12676)

  Show description for 75e9430
  

  LucasWilkinson

  and

  simon-mo

  authored
  75e9430

  Copy full SHA for 75e9430

• [Core] add and implement VLLM_LOGITS_PROCESSOR_THREADS (#12368)

  Show description for b3a0d01
  akeshet

  authored
  b3a0d01

  Copy full SHA for b3a0d01

• [ROCM][AMD][TRITON] Halving warps number for fw_prefill to reduce spilling (#12713)

  Show description for 64862d1
  

  maleksan85

  and

  Aleksandr Malyshev

  authored
  64862d1

  Copy full SHA for 64862d1

• Refactor Linear handling in TransformersModel (#12727)

  Show description for 249824c
  hmellor

  authored
  249824c

  Copy full SHA for 249824c

• [VLM] Add MLA with pure RoPE support for deepseek-vl2 models (#12729)

  Isotr0py

  authored
  98fd089

  Copy full SHA for 98fd089

• [Misc] Bump the compressed-tensors version (#12736)

  dsikka

  authored
  686006a

  Copy full SHA for 686006a

• [Model][Quant] Fix GLM, Fix fused module mappings for quantization (#12634)

  Show description for 7ff7a63
  

  kylesayrs

  and

  mgoin

  authored
  7ff7a63

  Copy full SHA for 7ff7a63

• [Doc] Update PR Reminder with link to Developer Slack (#12748)

  mgoin

  authored
  58b218d

  Copy full SHA for 58b218d

• [Bugfix] Fix OpenVINO model runner (#12750)

  hmellor

  authored
  fcf2e3d

  Copy full SHA for fcf2e3d

• [V1][Misc] Shorten FinishReason enum and use constant strings (#12760)

  njhill

  authored
  3d09e59

  Copy full SHA for 3d09e59

• [Doc] Remove performance warning for auto_awq.md (#12743)

  mgoin

  authored
  c53dc46

  Copy full SHA for c53dc46

• [Bugfix] Fix 'ModuleNotFoundError: No module named 'intel_extension_for_pytorch'' for --tensor-parallel-size more than 1 (#12546)

  Akashcodes732

  authored
  022bcc7

  Copy full SHA for 022bcc7

• [core][distributed] exact ray placement control (#12732)

  Show description for bc1bdec
  youkaichao

  authored
  bc1bdec

  Copy full SHA for bc1bdec

• Merging PR #12536

  Show description for 4c3aac5
  heheda12345

  authored
  4c3aac5

  Copy full SHA for 4c3aac5

• [Hardware][Intel-Gaudi] Enable FusedSDPA support for Intel Gaudi (HPU)

  SanjuCSudhakaran

  authored
  af8486d

  Copy full SHA for af8486d

• Add: Support for Sparse24Bitmask Compressed Models

  rahul-tuli

  authored
  3b2005e

  Copy full SHA for 3b2005e

• [VLM] Use shared field to pass token ids to model

  DarkLight1337

  authored
  a4ce74c

  Copy full SHA for a4ce74c

• [Docs] Drop duplicate [source] links

  russellb

  authored
  9a5b155

  Copy full SHA for 9a5b155

• [VLM] Qwen2.5-VL

  ywang96

  authored
  bf3b79e

  Copy full SHA for bf3b79e

Commits on Feb 6, 2025

• [VLM] Update compatibility with transformers 4.49

  DarkLight1337

  authored
  75404d0

  Copy full SHA for 75404d0

• [ROCm][Kernel] Using the correct warp_size value

  gshtras

  authored
  5b19b93

  Copy full SHA for 5b19b93

• [Bugfix] Better FP8 supported defaults

  LucasWilkinson

  authored
  76abd0c

  Copy full SHA for 76abd0c

• [Misc][Easy] Remove the space from the file name

  houseroad

  authored
  9cdea30

  Copy full SHA for 9cdea30

• [Model] LoRA Support for Ultravox model (#11253)

  thedebugger

  authored
  d88506d

  Copy full SHA for d88506d

• [Bugfix] Fix the test_ultravox.py's license (#12806)

  Show description for 56534cd
  houseroad

  authored
  56534cd

  Copy full SHA for 56534cd

• Improve TransformersModel UX (#12785)

  hmellor

  authored
  1a6fcad

  Copy full SHA for 1a6fcad

• [Misc] Remove duplicated DeepSeek V2/V3 model definition (#12793)

  mgoin

  authored
  449d1bc

  Copy full SHA for 449d1bc

• [Misc] Improve error message for incorrect pynvml (#12809)

  Show description for 0408efc
  youkaichao

  authored
  0408efc

  Copy full SHA for 0408efc

• [Misc] Update w2 scale loading for GPTQMarlinMoE (#12757)

  dsikka

  authored
  7ca9934

  Copy full SHA for 7ca9934

• [Docs] Add Google Cloud Slides (#12814)

  simon-mo

  authored
  cefd56e

  Copy full SHA for cefd56e

• [Attention] Use FA3 for MLA on Hopper (#12807)

  Show description for c786e75
  LucasWilkinson

  authored
  c786e75

  Copy full SHA for c786e75

• [misc] Reduce number of config file requests to HuggingFace (#12797)

  Show description for e152f29
  

  khluu

  and

  EC2 Default User

  authored
  e152f29

  Copy full SHA for e152f29

• [Misc] Remove unnecessary decode call (#12833)

  DarkLight1337

  authored
  1e57b1e

  Copy full SHA for 1e57b1e

• [Kernel] Make rotary_embedding ops more flexible with input shape (#12777)

  Isotr0py

  authored
  85ac82d

  Copy full SHA for 85ac82d

• [torch.compile] PyTorch 2.6 and nightly compatibility (#12393)

  Show description for 09b95e3
  youkaichao

  authored
  09b95e3

  Copy full SHA for 09b95e3

• [Doc] double quote cmake package in build.inc.md (#12840)

  jitseklomp

  authored
  afe74f7

  Copy full SHA for afe74f7

• [Bugfix] Fix unsupported FA version check for Turing GPU (#12828)

  Isotr0py

  authored
  8108ac8

  Copy full SHA for 8108ac8

• [V1] LoRA Support (#10957)

  Show description for 467a96a
  

  varun-sundar-rabindranath

  and

  Varun Sundar Rabindranath

  authored
  467a96a

  Copy full SHA for 467a96a

• Add Bamba Model (#10909)

  Show description for aff4045
  

  fabianlim

  and

  tlrmchlsmth

  authored
  aff4045

  Copy full SHA for aff4045

• [MISC] Check space in the file names in the pre commit checks (#12804)

  Show description for 741429a
  houseroad

  authored
  741429a

  Copy full SHA for 741429a

Commits on Feb 7, 2025

• [misc] Revert # 12833 (#12857)

  Show description for b260782
  

  khluu

  and

  EC2 Default User

  authored
  b260782

  Copy full SHA for b260782

• [Bugfix] FA2 illegal memory access (#12848)

  LucasWilkinson

  authored
  ef533d2

  Copy full SHA for ef533d2

• Make vllm compatible with verl (#12824)

  Show description for 433c4a4
  

  ZSL98

  and

  zhangshulai

  authored
  433c4a4

  Copy full SHA for 433c4a4

• [Bugfix] Missing quant_config in deepseek embedding layer (#12836)

  SzymonOzog

  authored
  aa375dc

  Copy full SHA for aa375dc

• Prevent unecessary requests to huggingface hub (#12837)

  maxdebayser

  authored
  6e1fc61

  Copy full SHA for 6e1fc61

• [MISC][EASY] Break check file names into entry and args in the pre-commit hooks (#12880)

  Show description for 1918aa1
  houseroad

  authored
  1918aa1

  Copy full SHA for 1918aa1

• [Misc] Remove unnecessary detokenization in multimodal processing (#12868)

  DarkLight1337

  authored
  ce26b16

  Copy full SHA for ce26b16

• PR #12718 (#12718)

  garg-amit

  authored
  538fab9

  Copy full SHA for 538fab9

• [V1] Logprobs and prompt logprobs support (#9880)

  Show description for 0630d45
  

  3 peopleauthored
  0630d45

  Copy full SHA for 0630d45

• [ROCm] [Feature] [Doc] [Dockerfile] [BugFix] Support Per-Token-Activation Per-Channel-Weight FP8 Quantization Inferencing (#12501)

  tjtanaa

  authored
  eaa92d4

  Copy full SHA for eaa92d4

• [V1] LM Eval With Streaming Integration Tests (#11590)

  robertgshaw2-redhat

  authored
  932c6b7

  Copy full SHA for 932c6b7

Commits on Feb 8, 2025

• [Bugfix] Fix disagg hang caused by the prefill and decode communication issues (#12723)

  Show description for 45cbc49
  houseroad

  authored
  45cbc49

  Copy full SHA for 45cbc49

• [V1][Minor] Remove outdated comment (#12928)

  Show description for b21f0f9
  WoosukKwon

  authored
  b21f0f9

  Copy full SHA for b21f0f9

• [V1] Move KV block hashes from Request to KVCacheManager (#12922)

  Show description for 3243158
  WoosukKwon

  authored
  3243158

  Copy full SHA for 3243158

• [Bugfix] Fix Qwen2_5_VLForConditionalGeneration packed_modules_mapping (#12905)

  jeejeelee

  authored
  306923d

  Copy full SHA for 306923d

• [Misc] Fix typo in the example file (#12896)

  Show description for cc01223
  DK-DARKmatter

  authored
  cc01223

  Copy full SHA for cc01223

• [Bugfix] Fix multi-round chat error when mistral tokenizer is used (#12859)

  Show description for d01f66b
  

  zifeitong

  and

  DarkLight1337

  authored
  d01f66b

  Copy full SHA for d01f66b

• [bugfix] respect distributed_executor_backend in world_size=1 (#12934)

  Show description for 91dd8f7
  youkaichao

  authored
  91dd8f7

  Copy full SHA for 91dd8f7

• [Misc] Add offline test for disaggregated prefill (#12418)

  Shaoting-Feng

  authored
  e31498b

  Copy full SHA for e31498b

• [V1][Minor] Move cascade attn logic outside _prepare_inputs (#12943)

  Show description for 4ea48fb
  WoosukKwon

  authored
  4ea48fb

  Copy full SHA for 4ea48fb

• [Build] Make pypi install work on CPU platform (#12874)

  wangxiyuan

  authored
  407b553

  Copy full SHA for 407b553

• [Hardware][Intel-Gaudi] Enable long-contexts + LoRA support for Intel Gaudi (#12812)

  Show description for 2880e21
  SanjuCSudhakaran

  authored
  2880e21

  Copy full SHA for 2880e21

• [misc] Add LoRA to benchmark_serving (#12898)

  Show description for 7e18376
  

  varun-sundar-rabindranath

  and

  Varun Sundar Rabindranath

  authored
  7e18376

  Copy full SHA for 7e18376

• [Misc] Log time consumption on weight downloading (#12926)

  waltforme

  authored
  011e612

  Copy full SHA for 011e612

• [CI] Resolve transformers-neuronx version conflict (#12925)

  liangfu

  authored
  c45d398

  Copy full SHA for c45d398

• [Doc] Correct HF repository for TeleChat2 models (#12949)

  waltforme

  authored
  256a2d2

  Copy full SHA for 256a2d2

• [Misc] Add qwen2.5-vl BNB support (#12944)

  Isotr0py

  authored
  4c8dd12

  Copy full SHA for 4c8dd12

• [CI/Build] Auto-fix Markdown files (#12941)

  DarkLight1337

  authored
  8a69e0e

  Copy full SHA for 8a69e0e

• [Bugfix] Remove unused seq_group_metadata_list from ModelInputForGPU (#12935)

  Show description for 913df14
  ShangmingCai

  authored
  913df14

  Copy full SHA for 913df14

• [bugfix] fix early import of flash attention (#12959)

  Show description for fe743b7
  youkaichao

  authored
  fe743b7

  Copy full SHA for fe743b7

• [VLM] Merged multi-modal processor for GLM4V (#12449)

  Show description for 86222a3
  jeejeelee

  authored
  86222a3

  Copy full SHA for 86222a3

• [V1][Minor] Remove outdated comment (#12968)

  Show description for 870c374
  WoosukKwon

  authored
  870c374

  Copy full SHA for 870c374

• [RFC] [Mistral] FP8 format (#10130)

  Show description for d366ccc
  

  patrickvonplaten

  and

  mgoin

  authored
  d366ccc

  Copy full SHA for d366ccc

• [V1] Cache uses_mrope in GPUModelRunner (#12969)

  WoosukKwon

  authored
  24700c3

  Copy full SHA for 24700c3

Commits on Feb 9, 2025

• [core] port pynvml into vllm codebase (#12963)

  Show description for cf797aa
  youkaichao

  authored
  cf797aa

  Copy full SHA for cf797aa

• [MISC] Always import version library first in the vllm package (#12979)

  Show description for 29f1d47
  houseroad

  authored
  29f1d47

  Copy full SHA for 29f1d47

Commits on Feb 10, 2025

• [core] improve error handling when wake up from sleep mode (#12981)

  Show description for 59fff4a
  youkaichao

  authored
  59fff4a

  Copy full SHA for 59fff4a

• [core][rlhf] add colocate example for RLHF (#12984)

  Show description for aa0ca5e
  youkaichao

  authored
  aa0ca5e

  Copy full SHA for aa0ca5e

• [V1] Use msgpack for core request serialization (#12918)

  Show description for 67c4637
  njhill

  authored
  67c4637

  Copy full SHA for 67c4637

• Check if selected backend is None in get_attn_backend_cls() (#12975)

  Show description for 44607e0
  terrytangyuan

  authored
  44607e0

  Copy full SHA for 44607e0

• [core] fix sleep mode and pytorch checkpoint compatibility (#13001)

  Show description for b2496bb
  youkaichao

  authored
  b2496bb

  Copy full SHA for b2496bb

• [Doc] Add link to tool_choice tracking issue in tool_calling.md (#13003)

  Show description for 2431371
  terrytangyuan

  authored
  2431371

  Copy full SHA for 2431371

• [misc] Add retries with exponential backoff for HF file existence check (#13008)

  khluu

  authored
  fde7126

  Copy full SHA for fde7126

• [Bugfix] Clean up and fix multi-modal processors (#13012)

  Show description for 51f0b5f
  DarkLight1337

  authored
  51f0b5f

  Copy full SHA for 51f0b5f

• Fix seed parameter behavior in vLLM (#13007)

  Show description for 2ae8890
  SmartManoj

  authored
  2ae8890

  Copy full SHA for 2ae8890

• [Model] Ultravox Model: Support v0.5 Release (#12912)

  Show description for 08b2d84
  farzadab

  authored
  08b2d84

  Copy full SHA for 08b2d84

Commits on Feb 11, 2025

• [misc] Fix setup.py condition to avoid AMD from being mistaken with CPU (#13022)

  Show description for 91e8767
  khluu

  authored
  91e8767

  Copy full SHA for 91e8767

• [V1][Minor] Move scheduler outputs to a separate file (#13062)

  Show description for 2ff4857
  WoosukKwon

  authored
  2ff4857

  Copy full SHA for 2ff4857

• [Docs] Annouce Meta Meetup (#13065)

  Show description for 2c0f582
  simon-mo

  authored
  2c0f582

  Copy full SHA for 2c0f582

• [Bugfix] Support missing tool parameters in mistral tokenizer (#12884)

  Show description for cb080f3
  fgreinacher

  authored
  cb080f3

  Copy full SHA for cb080f3

• [Benchmark] Add BurstGPT to benchmark_serving (#13063)

  Show description for 58047c6
  

  WoosukKwon

  and

  ywang96

  authored
  58047c6

  Copy full SHA for 58047c6

• [Core] Don't do platform detection at import time (#12933)

  Show description for c320ca8
  russellb

  authored
  c320ca8

  Copy full SHA for c320ca8

• [Misc] LoRA - Refactor Punica ops tests (#12970)

  Show description for 78a141d
  

  varun-sundar-rabindranath

  and

  Varun Sundar Rabindranath

  authored
  78a141d

  Copy full SHA for 78a141d

• [Bugfix]: Reasoning output bug according to the chat template change (#13025)

  Show description for fc6485d
  gaocegege

  authored
  fc6485d

  Copy full SHA for fc6485d

• [V1][Metrics] Add GPU prefix cache hit rate % gauge (#12592)

  comaniac

  authored
  41c5dd4

  Copy full SHA for 41c5dd4

• [executor] init local_rank as device index (#13027)

  Show description for 9cf4759
  MengqingCao

  authored
  9cf4759

  Copy full SHA for 9cf4759

• [ROCm] Using a more precise memory profiling (#12624)

  Show description for 7539bbc
  gshtras

  authored
  7539bbc

  Copy full SHA for 7539bbc

• [Build] Fix cuda link target of cumem_allocator in CPU env (#12863)

  Show description for da31719
  

  guoyuhong

  and

  tlrmchlsmth

  authored
  da31719

  Copy full SHA for da31719

• [Platform] add pre_register_and_update function (#12432)

  Show description for 2e3b969
  wangxiyuan

  authored
  2e3b969

  Copy full SHA for 2e3b969

• [Bugfix] fix flaky test (#13089)

  Show description for 110f59a
  SmartManoj

  authored
  110f59a

  Copy full SHA for 110f59a

• [V1][Metrics] Add several request timing histograms (#12644)

  Show description for 75e6e14
  markmc

  authored
  75e6e14

  Copy full SHA for 75e6e14

• Set torch_dtype in TransformersModel (#13088)

  Show description for ad97763
  hmellor

  authored
  ad97763

  Copy full SHA for ad97763

• [Misc] Fix typo at comments at metrics.py (#13024)

  je1lee

  authored
  bf3e052

  Copy full SHA for bf3e052

• [Bugfix] Do not use resource module on Windows (#12858) (#13029)

  MoonRide303

  authored
  21f5d50

  Copy full SHA for 21f5d50

• [BugFix] Pop instead of del CUDA_VISIBLE_DEVICES (#12962)

  Show description for 6c4dbe2
  HollowMan6

  authored
  6c4dbe2

  Copy full SHA for 6c4dbe2

• Fix initializing GGUF weights for ColumnParallelLinear when using tensor parallel > 1 (#13023)

  SzymonOzog

  authored
  2b25b7d

  Copy full SHA for 2b25b7d

• [CI/Build][Bugfix] Fix CPU backend default threads num (#13077)

  bigPYJ1151

  authored
  565c1ef

  Copy full SHA for 565c1ef

• [Doc] Improve OpenVINO installation doc (#13102)

  Show description for deb6c1c
  hmellor

  authored
  deb6c1c

  Copy full SHA for deb6c1c

• [Bugfix] Guided decoding falls back to outlines when fails to import xgrammar (#12976)

  Show description for 14ecab5
  terrytangyuan

  authored
  14ecab5

  Copy full SHA for 14ecab5

• [Misc] Move pre-commit suggestion back to the end (#13114)

  Show description for 72c2b68
  russellb

  authored
  72c2b68

  Copy full SHA for 72c2b68