Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use FlashAttention for multi_query_kv_attention #4

Merged
merged 8 commits into from
Mar 2, 2023
Merged

Use FlashAttention for multi_query_kv_attention #4

merged 8 commits into from
Mar 2, 2023

Conversation

WoosukKwon
Copy link
Collaborator

@WoosukKwon WoosukKwon commented Mar 2, 2023 •
edited
Loading

This PR is to use FlashAttention kernels for multi_query_kv_attention, which performs masked attention for the prompt inputs.

Pros

  • FlashAttention is fast and memory-efficient.
  • FlashAttention supports 1D inputs and only invokes a single kernel to handle multiple sequences with variable lengths.

Cons

Besides, note that FlashAttention does not support cached KV, which is required for interactive generation.

Tested models:

  • OPT-125M
  • OPT-350M
  • OPT-1.3B
  • OPT-2.7B
  • OPT-6.7B
  • OPT-13B

Tested GPUs:

  • A100

@WoosukKwon WoosukKwon merged commit 3e9f991 into main Mar 2, 2023
@WoosukKwon WoosukKwon deleted the flash-attn branch March 2, 2023 05:13
@TheBloke TheBloke mentioned this pull request Jul 20, 2023
Closed
@tmm1 tmm1 mentioned this pull request Aug 3, 2023
Closed
xiangyuT added a commit to xiangyuT/vllm that referenced this pull request Oct 24, 2023
@xiangyuT
Add BigDL Llama worker for batching on decoding (vllm-project#4)
02b4cac 
* Init

* refine
@shanshanpt shanshanpt mentioned this pull request Nov 17, 2023
Closed
@junior-zsy junior-zsy mentioned this pull request Nov 20, 2023
Closed
hongxiayang pushed a commit to hongxiayang/vllm that referenced this pull request Feb 13, 2024
@WoosukKwon
Use FlashAttention for multi_query_kv_attention (vllm-project#4)
e7c912b
luo-cheng2021 pushed a commit to luo-cheng2021/vllm that referenced this pull request Mar 12, 2024
@slyalin
Merge pull request vllm-project#4 from slyalin/optimum_models
8a9862f 
Support for optimum-intel models
luo-cheng2021 pushed a commit to luo-cheng2021/vllm that referenced this pull request Mar 25, 2024
@ilya-lavrenov
Merge pull request vllm-project#4 from luo-cheng2021/luocheng/openvin…
658407a 
…o-model-executor

Adapt OpenVINO CPU plugin implementation
mzusman pushed a commit to mzusman/vllm that referenced this pull request Apr 16, 2024
Merged in jamba-3 (pull request vllm-project#4)
0330e14 
BA-78760: Jamba

* Add support for n concat and splitting

* change naming

* input_metadata is a dict list now in order to pass "n"

* clean up code from unecessary changes and prints

* Remove kv cache allocation in case of mamba layer

* Add the considerations of mamba layer cache into the num of blocks
calculation

* Delete mamba cache after profile

* Remove prints

* Cleaning

* - and not _ for requirements

Approved-by: Tomer Asida
@dlopes78 dlopes78 mentioned this pull request May 8, 2024
Open
linxihui added a commit to linxihui/vllm that referenced this pull request May 14, 2024
@linxihui
Merge pull request vllm-project#4 from beagleski/bapatra/patching-for-su
7646e00 
patching for having type su
@Alexei-V-Ivanov-AMD Alexei-V-Ivanov-AMD mentioned this pull request May 16, 2024
Merged
@afeldman-nm afeldman-nm mentioned this pull request May 21, 2024
Merged
@yuhuixu1993 yuhuixu1993 mentioned this pull request Jun 2, 2024
Open
@afeldman-nm afeldman-nm mentioned this pull request Jun 3, 2024
Closed
yukavio pushed a commit to yukavio/vllm that referenced this pull request Jul 3, 2024
@afeldman-nm @tlrmchlsmth
Semi-structured 2:4 sparsity via SparseSemiStructuredTensor vllm-proj…
4f8d12e 
…ect#4

magic_wand semi_structured_sparse_tensor_linear branch integrates 2:4 semi-structured sparsity into SparseTensor. This PR adds a new sparsity config for 2:4 sparsity to neuralmagic-vllm, using the SparseTensor 2:4 support.

This PR also refactors the sparse linear method into a separate file, vllm/model_executor/layers/sparsity/sparse_w16a16_linear_method.py, which supports all sparsity formats.
yukavio pushed a commit to yukavio/vllm that referenced this pull request Jul 3, 2024
@afeldman-nm @robertgshaw2-neuralmagic
Semi-structured 2:4 sparsity via SparseSemiStructuredTensor vllm-proj…
81dba47 
…ect#4

magic_wand semi_structured_sparse_tensor_linear branch integrates 2:4 semi-structured sparsity into SparseTensor. This PR adds a new sparsity config for 2:4 sparsity to neuralmagic-vllm, using the SparseTensor 2:4 support.

This PR also refactors the sparse linear method into a separate file, vllm/model_executor/layers/sparsity/sparse_w16a16_linear_method.py, which supports all sparsity formats.
@oliver-li oliver-li mentioned this pull request Jul 5, 2024
Open
@haichuan1221 haichuan1221 mentioned this pull request Jul 5, 2024
Merged
@haichuan1221 haichuan1221 mentioned this pull request Jul 8, 2024
Open
@ehuaa ehuaa mentioned this pull request Jul 19, 2024
Closed
@xinzaifeixiang1992 xinzaifeixiang1992 mentioned this pull request Jul 24, 2024
Open
@alixiaodi alixiaodi mentioned this pull request Aug 2, 2024
Closed
@Minami-su Minami-su mentioned this pull request Aug 11, 2024
Closed
@wangwensuo wangwensuo mentioned this pull request Aug 22, 2024
Open
zeroorhero pushed a commit to zeroorhero/vllm that referenced this pull request Sep 23, 2024
@KuntaiDu
Merge pull request vllm-project#4 from KuntaiDu/yihua-lookup-buffer
c5b7232 
Lookup buffer implementation
@liulisi16323 liulisi16323 mentioned this pull request Sep 24, 2024
Open
1 task
@Clint-chan Clint-chan mentioned this pull request Sep 29, 2024
Open
1 task
@SpaceHunterInf SpaceHunterInf mentioned this pull request Sep 30, 2024
Closed
1 task
This was referenced Oct 12, 2024
Closed
Open
@alexchenyu alexchenyu mentioned this pull request Oct 13, 2024
Open
1 task
@xxzhang0927 xxzhang0927 mentioned this pull request Oct 30, 2024
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@WoosukKwon