Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement single_query_cached_kv_attention kernel #3

Merged
merged 14 commits into from
Mar 1, 2023
Merged

Implement single_query_cached_kv_attention kernel #3

merged 14 commits into from
Mar 1, 2023

Conversation

WoosukKwon
Copy link
Collaborator

This PR adds the single_query_cached_kv_attention kernel.

Supported data types:

  • half
  • float

Tested models:

  • OPT-125M
  • OPT-350M
  • OPT-1.3B
  • OPT-2.7B
  • OPT-6.7B
  • OPT-13B

Tested GPUs:

  • A100

@WoosukKwon WoosukKwon merged commit 0deacbc into main Mar 1, 2023
@WoosukKwon WoosukKwon deleted the attention-kernel branch March 1, 2023 23:11
@murongweibo murongweibo mentioned this pull request Jul 11, 2023
Closed
@TheBloke TheBloke mentioned this pull request Jul 20, 2023
Closed
v1nc3nt27 pushed a commit to v1nc3nt27/vllm that referenced this pull request Sep 12, 2023
@ri938
Merge pull request vllm-project#3 from ri938/merge_linear_layers
a3ac858 
Merge linear layers
xiangyuT pushed a commit to xiangyuT/vllm that referenced this pull request Oct 18, 2023
@gc-fu
Send multiple requests to underlying model (vllm-project#3)
e36fc39 
* add multiple requests test

* fix
@shanshanpt shanshanpt mentioned this pull request Nov 17, 2023
Closed
@junior-zsy junior-zsy mentioned this pull request Nov 20, 2023
Closed
hongxiayang pushed a commit to hongxiayang/vllm that referenced this pull request Feb 13, 2024
@WoosukKwon
Implement single_query_cached_kv_attention kernel (vllm-project#3)
98a4895
Spycsh pushed a commit to Spycsh/vllm that referenced this pull request Feb 27, 2024
@mdvoretc-intel
Enable cache ops for beam search (vllm-project#3)
f51f149 
Co-authored-by: Mikhail Dvoretckii <mdvoretckii@habana.ai>
@Qubitium Qubitium mentioned this pull request Mar 5, 2024
Open
luo-cheng2021 pushed a commit to luo-cheng2021/vllm that referenced this pull request Mar 12, 2024
@slyalin
Merge pull request vllm-project#3 from slyalin/window_and_alibi
ce80498 
Passing alibi_slopes and sliding_window to PagedAttention extension
luo-cheng2021 pushed a commit to luo-cheng2021/vllm that referenced this pull request Mar 20, 2024
@ilya-lavrenov
Merge pull request vllm-project#3 from ilya-lavrenov/docker-file
354ca31 
Added dockerfile with vLLM + openvino
mujjingun added a commit to gmlwns2000/vllm-timber that referenced this pull request Apr 15, 2024
@mujjingun
Merge pull request vllm-project#3 from DeepAuto-AI/geon-dev
52ed876 
Geon dev
mzusman added a commit to mzusman/vllm that referenced this pull request Apr 16, 2024
@mzusman @ErezSC42 @tomeras91
Jamba mamba (vllm-project#3)
07cc899 
* Remove assertion

* adapting jamba vllm to changes after hf release, working on weight loading in modeling file

* splitting the JambaDecoderLayer to JambaMambaDecoderLayer and JambaAttentionDecoderLayer

* weight loading from hf checkpoint supposedly works, might be a mixup in the MoE between the gated and non-gated weights

* Add mamba from jamba modeling file

* Remove slow forward

* Modifications to mamba_mixer

* Save changes, WIP

* Fix cache placement

* Debugging

* Additions and logging

* Jamba with mamba cache handling

* Clean up

* Another cleanup

* Use vllm's RMSNorm instead of JambaRMSNorm, Thier implementation is with
fused kernel

* Clean up and orginization of the objects to handle the mamba cache

* Shorten the code for kv cache mem

* Move cache handling inside the Mixer

* Add mamba to the wheel requirements

* Add mamba to the requirements script

* Add mamba_metadata

* Add to __init__ __all__

* Revert 2 commits

ad1a3db 'Add mamba to the requirements script'
75ed2c8 'Add mamba to the wheel requirements'

* Clean up

* Naming

* Apply whitespace suggestions from code review

* pass tie_word_embeddings to PretrainedConfig init

* Replace repeat with expand as expand doesn't require more mem

* Allocate really small cache if needed , don't use meta

* Fix for expanded

---------

Co-authored-by: Mor Zusman <morz@ai21.com>
Co-authored-by: Erez Schwartz <erezs@ai21.com>
Co-authored-by: tomeras91 <57313761+tomeras91@users.noreply.github.com>
sfc-gh-hazhang referenced this pull request in sfc-gh-hazhang/vllm May 7, 2024
@iamontheinet
Merge pull request #3 from Snowflake-Labs/arctic-linting
bfe5011
@dlopes78 dlopes78 mentioned this pull request May 8, 2024
Open
linxihui pushed a commit to linxihui/vllm that referenced this pull request May 14, 2024
@codedecde
Merge pull request vllm-project#3 from beagleski/bapatra/bugfix-longr…
561d5a8 
…ope-type

minor change for LongRoPE config to account for rename from longrope …
@Alexei-V-Ivanov-AMD Alexei-V-Ivanov-AMD mentioned this pull request May 15, 2024
Merged
@yuhuixu1993 yuhuixu1993 mentioned this pull request Jun 2, 2024
Open
@afeldman-nm afeldman-nm mentioned this pull request Jun 3, 2024
Closed
@afeldman-nm afeldman-nm mentioned this pull request Jun 21, 2024
Closed
@robertgshaw2-neuralmagic robertgshaw2-neuralmagic mentioned this pull request Jun 28, 2024
Open
@Juelianqvq Juelianqvq mentioned this pull request Jul 3, 2024
Closed
@oliver-li oliver-li mentioned this pull request Jul 5, 2024
Open
This was referenced Jul 5, 2024
Merged
Open
@ehuaa ehuaa mentioned this pull request Jul 19, 2024
Closed
@xinzaifeixiang1992 xinzaifeixiang1992 mentioned this pull request Jul 24, 2024
Open
@alixiaodi alixiaodi mentioned this pull request Aug 2, 2024
Closed
@Minami-su Minami-su mentioned this pull request Aug 11, 2024
Closed
@wangwensuo wangwensuo mentioned this pull request Aug 22, 2024
Open
@mgoin mgoin mentioned this pull request Sep 16, 2024
Merged
PanJason added a commit to PanJason/vllm that referenced this pull request Sep 21, 2024
@PanJason @TKONIY
Add memory tiering for CC (vllm-project#3)
989059e 
* Add example disk swap config. Add unit tests for CC with memory tiering

* Layered transfer for DRAM. Transfer in cuda streams

* Fix the missing arg

* Fix context caching online serving

This commit enables layered transmission for DRAM first. Now the
transmission is done in different cuda streams. xformers, flash infer
and flash attention are supported. Optimized transfer for disk
is still pending.

Cherry-pick Yangshen's commit

---------

Co-authored-by: yangshen <yangshen.d@outlook.com>
PanJason added a commit to PanJason/vllm that referenced this pull request Sep 21, 2024
@PanJason @TKONIY
Add memory tiering for CC (vllm-project#3)
f1123cd 
* Add example disk swap config. Add unit tests for CC with memory tiering

* Layered transfer for DRAM. Transfer in cuda streams

* Fix the missing arg

* Fix context caching online serving

This commit enables layered transmission for DRAM first. Now the
transmission is done in different cuda streams. xformers, flash infer
and flash attention are supported. Optimized transfer for disk
is still pending.

Cherry-pick Yangshen's commit

---------

Co-authored-by: yangshen <yangshen.d@outlook.com>
zeroorhero pushed a commit to zeroorhero/vllm that referenced this pull request Sep 23, 2024
@KuntaiDu
Merge pull request vllm-project#3 from KuntaiDu/yihua-kv-pipe
1377912 
Optimize the KV transfer pipe implementation
@liulisi16323 liulisi16323 mentioned this pull request Sep 24, 2024
Open
1 task
@Clint-chan Clint-chan mentioned this pull request Sep 29, 2024
Open
1 task
@SpaceHunterInf SpaceHunterInf mentioned this pull request Sep 30, 2024
Closed
1 task
This was referenced Oct 12, 2024
Closed
Open
@alexchenyu alexchenyu mentioned this pull request Oct 13, 2024
Open
1 task
@mgoin mgoin mentioned this pull request Oct 21, 2024
Merged
@xxzhang0927 xxzhang0927 mentioned this pull request Oct 30, 2024
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@WoosukKwon