Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🚨All attention refactor🚨 #35235

Merged
merged 99 commits into from
Dec 18, 2024
Merged

🚨All attention refactor🚨 #35235

merged 99 commits into from
Dec 18, 2024
+5,635 −9,778

Conversation

ArthurZucker
Copy link
Collaborator

@ArthurZucker ArthurZucker commented Dec 12, 2024 •
edited by Cyrilvallez
Loading

What does this PR do?

Todo in this PR:

  • Cohere
  • Chameleon
  • DBRX
  • Gemma
  • Gemma2
  • GLM (modular donc rien à faire je crois)
  • gpt_neoX et GPT2
  • Granite
  • Jamba
  • JetMoe
  • Mimi
  • Mistral
  • Mixtral
  • Mllama
  • Moshi
  • Nemotron
  • OPT
  • Phi
  • Ph3
  • PhiMoe
  • Qwen2
  • qwen2Moe
  • qwen2VL
  • SableML
  • StartCoder2 -> Modular normalement oK
  • Idefics1,2,3
  • Olmo
  • Olmo2
  • Siglip
  • Whisper

@ArthurZucker ArthurZucker force-pushed the all-attention-refactor branch from 0dc9253 to d1aa9ce Compare December 12, 2024 13:49
ArthurZucker
ArthurZucker commented Dec 13, 2024
View reviewed changes
src/transformers/modeling_utils.py Outdated
)


class GradientCheckpointLayer(torch.nn.Module):
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should help with kwargs as well

@ArthurZucker ArthurZucker mentioned this pull request Dec 16, 2024
Merged
@Cyrilvallez Cyrilvallez force-pushed the all-attention-refactor branch from 8b56823 to ecd814b Compare December 16, 2024 11:28
@poedator
Copy link
Contributor

poedator commented Jan 14, 2025 •
edited
Loading

My friends use a GPT2Model in production and want to compile it with StaticCache. With the maintainers blessing, I would try to create a PR with DynamicCache / StaticCache support in GPT2Model.
I am quite familiar with Cache class, I already coded some and made the DynamicCache work.

Please let me know if there are any hidden obstacles in Cache implementation for GPT2? Which tests to run or add?
@ArthurZucker

@Rocketknight1
Copy link
Member

cc @gante to that question!

@gante
Copy link
Member

gante commented Jan 15, 2025

I've chatted to @poedator offline -- I couldn't think of any obstacle in particular, and suggested a) to ensure we leave a deprecation warning regarding the old cache format b) use RUN_SLOW=1 py.test tests/models/gpt2/test_modeling_gpt2.py as a correctness check (gpt2 is fairly well tested, especially wrt text generation)

@poedator
Copy link
Contributor

It looks like test_flash_attn_2_from_config is broken - it expects attention layer to have flashattention in its name,
if "FlashAttention" in module.__class__.__name__:...
but after this refactoring, the attention classes are named differently.

ref

transformers/tests/test_modeling_common.py

Line 4641 in 5fa3534

if "FlashAttention" in module.__class__.__name__:

please fix or suspend the test.
@ArthurZucker

@poedator poedator mentioned this pull request Jan 18, 2025
Open
@ArthurZucker
Copy link
Collaborator Author

indeed gimme a min!

@imangohari1 imangohari1 mentioned this pull request Jan 31, 2025
Merged
3 tasks
@Miking98 Miking98 mentioned this pull request Feb 2, 2025
Closed
youkaichao pushed a commit to vllm-project/vllm that referenced this pull request Feb 3, 2025
@ArthurZucker @Isotr0py
@hmellor @DarkLight1337 @mgoin
[Model]: Add transformers backend support (#11330)
a1a2aaa 
# Adds support for `transformers` as a backend

Following huggingface/transformers#35235, a
bunch of models should already be supported, we are ramping up support
for more models.

Thanks @Isotr0py for the TP support, and @hmellor for his help as well!
This includes: 
- `trust_remote_code=True` support: any model on the hub, if it
implements attention the correct way can be natively supported!!
- tensor parallel support

---------

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <41363108+Isotr0py@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@ydshieh ydshieh mentioned this pull request Feb 4, 2025
Merged
ydshieh added a commit that referenced this pull request Feb 4, 2025
@ydshieh
Update tests regarding attention types after #35235 (#36024)
fe52679 
* update

* update

* update

* dev-ci

* more changes

* fix

* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
@Rocketknight1 Rocketknight1 mentioned this pull request Feb 4, 2025
Closed
4 tasks
@wizeng23 wizeng23 mentioned this pull request Feb 4, 2025
Merged
4 tasks
This was referenced Feb 5, 2025
Closed
Open
@ydshieh ydshieh mentioned this pull request Feb 5, 2025
Open
siqi654321 pushed a commit to siqi654321/DeepSpeed that referenced this pull request Feb 7, 2025
@loadams
Pin nv-a6000 workflow (deepspeedai#6938)
7059fe9 
Breaking change in transformers is
huggingface/transformers#35235. Need to make
changes to unpin nv-a6000 workflow.

Signed-off-by: siqi <siqi@tecorigin.com>
fxmarty-amd pushed a commit to fxmarty-amd/vllm that referenced this pull request Feb 7, 2025
@ArthurZucker @Isotr0py
@hmellor @DarkLight1337 @mgoin @fxmarty-amd
[Model]: Add transformers backend support (vllm-project#11330)
3d5064e 
# Adds support for `transformers` as a backend

Following huggingface/transformers#35235, a
bunch of models should already be supported, we are ramping up support
for more models.

Thanks @Isotr0py for the TP support, and @hmellor for his help as well!
This includes:
- `trust_remote_code=True` support: any model on the hub, if it
implements attention the correct way can be natively supported!!
- tensor parallel support

---------

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <41363108+Isotr0py@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: Felix Marty <felmarty@amd.com>
traincheck-team pushed a commit to traincheck-team/DeepSpeed that referenced this pull request Feb 9, 2025
@loadams @traincheck-team
Pin nv-a6000 workflow (deepspeedai#6938)
6f2bbf7 
Breaking change in transformers is
huggingface/transformers#35235. Need to make
changes to unpin nv-a6000 workflow.
ShangmingCai pushed a commit to ShangmingCai/vllm that referenced this pull request Feb 10, 2025
@ArthurZucker @Isotr0py
@hmellor @DarkLight1337 @mgoin @ShangmingCai
[Model]: Add transformers backend support (vllm-project#11330)
240493e 
# Adds support for `transformers` as a backend

Following huggingface/transformers#35235, a
bunch of models should already be supported, we are ramping up support
for more models.

Thanks @Isotr0py for the TP support, and @hmellor for his help as well!
This includes: 
- `trust_remote_code=True` support: any model on the hub, if it
implements attention the correct way can be natively supported!!
- tensor parallel support

---------

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <41363108+Isotr0py@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
This was referenced Feb 12, 2025
Closed
Draft
elvircrn pushed a commit to elvircrn/transformers that referenced this pull request Feb 13, 2025
@ydshieh @elvircrn
Update tests regarding attention types after huggingface#35235 (huggi…
8021eb7 
…ngface#36024)

* update

* update

* update

* dev-ci

* more changes

* fix

* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
@hlky hlky mentioned this pull request Feb 14, 2025
Draft
sbucaille pushed a commit to sbucaille/transformers that referenced this pull request Feb 16, 2025
@ydshieh @sbucaille
Update tests regarding attention types after huggingface#35235 (huggi…
a075f01 
…ngface#36024)

* update

* update

* update

* dev-ci

* more changes

* fix

* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
gyou2021 pushed a commit to gyou2021/DeepSpeed that referenced this pull request Feb 18, 2025
@loadams @gyou2021
Pin nv-a6000 workflow (deepspeedai#6938)
68066af 
Breaking change in transformers is
huggingface/transformers#35235. Need to make
changes to unpin nv-a6000 workflow.

Signed-off-by: gyou2021 <ganmei.you@intel.com>
gyou2021 pushed a commit to gyou2021/DeepSpeed that referenced this pull request Feb 18, 2025
@loadams @gyou2021
Pin nv-a6000 workflow (deepspeedai#6938)
049ed93 
Breaking change in transformers is
huggingface/transformers#35235. Need to make
changes to unpin nv-a6000 workflow.

Signed-off-by: gyou2021 <ganmei.you@intel.com>
panf2333 pushed a commit to yottalabsai/vllm that referenced this pull request Feb 18, 2025
@ArthurZucker @Isotr0py
@hmellor @DarkLight1337 @mgoin @panf2333
[Model]: Add transformers backend support (vllm-project#11330)
9884f10 
# Adds support for `transformers` as a backend

Following huggingface/transformers#35235, a
bunch of models should already be supported, we are ramping up support
for more models.

Thanks @Isotr0py for the TP support, and @hmellor for his help as well!
This includes: 
- `trust_remote_code=True` support: any model on the hub, if it
implements attention the correct way can be natively supported!!
- tensor parallel support

---------

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <41363108+Isotr0py@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
@kevinstephano kevinstephano mentioned this pull request Feb 20, 2025
Open
kerthcet pushed a commit to kerthcet/vllm that referenced this pull request Feb 21, 2025
@ArthurZucker @Isotr0py
@hmellor @DarkLight1337 @mgoin @kerthcet
[Model]: Add transformers backend support (vllm-project#11330)
d5e0491 
# Adds support for `transformers` as a backend

Following huggingface/transformers#35235, a
bunch of models should already be supported, we are ramping up support
for more models.

Thanks @Isotr0py for the TP support, and @hmellor for his help as well!
This includes: 
- `trust_remote_code=True` support: any model on the hub, if it
implements attention the correct way can be natively supported!!
- tensor parallel support

---------

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <41363108+Isotr0py@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Cyrilvallez Cyrilvallez Cyrilvallez left review comments

@vasqu vasqu vasqu left review comments

@LysandreJik LysandreJik LysandreJik approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.