Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add intel xpu support for TGI #1475

Merged
merged 10 commits into from
Apr 26, 2024
Merged

add intel xpu support for TGI #1475

merged 10 commits into from
Apr 26, 2024

Conversation

sywangyi
Copy link
Contributor

What does this PR do?

Fixes # (issue)

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sywangyi sywangyi marked this pull request as draft January 24, 2024 02:00
OlivierDehaene
OlivierDehaene reviewed Jan 26, 2024
View reviewed changes
Copy link
Member

@OlivierDehaene OlivierDehaene left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great!

server/text_generation_server/models/flash_causal_lm.py Show resolved Hide resolved
server/text_generation_server/utils/dist.py Show resolved Hide resolved
server/text_generation_server/utils/flash_attn.py Show resolved Hide resolved
server/text_generation_server/utils/layers.py Show resolved Hide resolved
@Narsil
Copy link
Collaborator

Narsil commented Jan 26, 2024

Overall looks pretty good !

I think we'll move to propre enum SYSTEM instead of IS_XXXX since there's no way a user could be running 2 devices simultaneously. But we'll take care of that in a subsequent PR if that's OK.

@sywangyi
Copy link
Contributor Author

thanks very much @Narsil and @OlivierDehaene for your comment. adding @yao-matrix

@sywangyi
Copy link
Contributor Author

sywangyi commented Feb 1, 2024

@Narsil @OlivierDehaene I add xpu smi in env runtime, do you think is it proper to add this? this is to dump intel XPU version.

@Narsil
Copy link
Collaborator

Narsil commented Feb 1, 2024

@Narsil @OlivierDehaene I add xpu smi in env runtime, do you think is it proper to add this? this is to dump intel XPU version.

It's OK in the current iteration (since showing N/A on most systems).

@Narsil
Copy link
Collaborator

Narsil commented Feb 1, 2024

Feel free to open up the draft whenever you're OK so we can run the tests + merge.

@github-actions github-actions bot added the Stale label Mar 3, 2024
@sywangyi sywangyi marked this pull request as ready for review March 8, 2024 02:48
@github-actions github-actions bot removed the Stale label Mar 9, 2024
@sywangyi
Copy link
Contributor Author

@Narsil could you review the PR？done by my side.

@sywangyi
Copy link
Contributor Author

@Narsil @mfuntowicz CI node is ready, could we move on?

@yao-matrix
Copy link

@Narsil @mfuntowicz , pls feel free to let us know any issues, since CI is ready, I think we can launch CI and merge this PR, thx.

cc @kding1

sywangyi and others added 9 commits April 26, 2024 13:41
@sywangyi @Narsil
add intel xpu support for TGI
49cd0ce 
Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
@sywangyi @Narsil
fix review comments
bc069db 
Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
@sywangyi @Narsil
add xpu smi support in env runtime
515a0ed 
Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
@sywangyi @Narsil
align to ipex llm ops
23a1cb0 
Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
@sywangyi @Narsil
update docker file
02537ec 
Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
@sywangyi @Narsil
update the API and dockerfile
0343a4b 
Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
@sywangyi @Narsil
no requirements_common.txt, update dockerfile
878101e 
Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
@sywangyi @Narsil
re-enable xpu
8815379 
Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
@mfuntowicz @Narsil
Add build-and-push-image for Intel GPUs
b67ce71
@Narsil Narsil merged commit 45ecf9d into huggingface:main Apr 26, 2024
3 of 5 checks passed
martinigoyanes pushed a commit to martinigoyanes/text-generation-inference that referenced this pull request Apr 30, 2024
@sywangyi @mfuntowicz
@Narsil @martinigoyanes
add intel xpu support for TGI (huggingface#1475)
08f92bd 
# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet
though.

Once merged, your PR is going to appear in the release notes with the
title you set, so make sure it's a great title that fully reflects the
extent of your awesome contribution.

Then, please replace this with a description of the change and which
issue is fixed (if applicable). Please also include relevant motivation
and context. List any dependencies (if any) that are required for this
change.

Once you're done, someone will review your PR shortly (see the section
"Who can review?" below to tag some potential reviewers). They may
suggest changes to make the code even better. If no one reviewed your PR
after a week has passed, don't hesitate to post a new comment
@-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the
other checks if that's the case).
- [ ] Did you read the [contributor
guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the
[forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes?
Here are the
[documentation
guidelines](https://github.com/huggingface/transformers/tree/main/docs),
and
[here are tips on formatting
docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have
passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the
right person to tag with @


@OlivierDehaene OR @Narsil

 -->

---------

Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
Co-authored-by: Morgan Funtowicz <funtowiczmo@gmail.com>
Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
Nilabhra pushed a commit to TII-AI-Research-Center/text-generation-inference that referenced this pull request May 14, 2024
@sywangyi @mfuntowicz
@Narsil @Nilabhra
add intel xpu support for TGI (huggingface#1475)
83cda09 
# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet
though.

Once merged, your PR is going to appear in the release notes with the
title you set, so make sure it's a great title that fully reflects the
extent of your awesome contribution.

Then, please replace this with a description of the change and which
issue is fixed (if applicable). Please also include relevant motivation
and context. List any dependencies (if any) that are required for this
change.

Once you're done, someone will review your PR shortly (see the section
"Who can review?" below to tag some potential reviewers). They may
suggest changes to make the code even better. If no one reviewed your PR
after a week has passed, don't hesitate to post a new comment
@-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the
other checks if that's the case).
- [ ] Did you read the [contributor
guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the
[forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes?
Here are the
[documentation
guidelines](https://github.com/huggingface/transformers/tree/main/docs),
and
[here are tips on formatting
docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have
passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the
right person to tag with @


@OlivierDehaene OR @Narsil

 -->

---------

Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
Co-authored-by: Morgan Funtowicz <funtowiczmo@gmail.com>
Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
@sywangyi sywangyi deleted the xpu_support_pr branch June 5, 2024 00:36
kdamaszk pushed a commit to kdamaszk/tgi-gaudi that referenced this pull request Jun 10, 2024
@sywangyi @mfuntowicz
@Narsil @kdamaszk
add intel xpu support for TGI (huggingface#1475)
561f2b1 
<!--
Congratulations! You've made it this far! You're not quite done yet
though.

Once merged, your PR is going to appear in the release notes with the
title you set, so make sure it's a great title that fully reflects the
extent of your awesome contribution.

Then, please replace this with a description of the change and which
issue is fixed (if applicable). Please also include relevant motivation
and context. List any dependencies (if any) that are required for this
change.

Once you're done, someone will review your PR shortly (see the section
"Who can review?" below to tag some potential reviewers). They may
suggest changes to make the code even better. If no one reviewed your PR
after a week has passed, don't hesitate to post a new comment
@-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

- [ ] This PR fixes a typo or improves the docs (you can dismiss the
other checks if that's the case).
- [ ] Did you read the [contributor
guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the
[forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes?
Here are the
[documentation
guidelines](https://github.com/huggingface/transformers/tree/main/docs),
and
[here are tips on formatting
docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Anyone in the community is free to review the PR once the tests have
passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the
right person to tag with @

@OlivierDehaene OR @Narsil

 -->

---------

Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
Co-authored-by: Morgan Funtowicz <funtowiczmo@gmail.com>
Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
kdamaszk pushed a commit to kdamaszk/tgi-gaudi that referenced this pull request Jun 10, 2024
@sywangyi @mfuntowicz
@Narsil @kdamaszk
add intel xpu support for TGI (huggingface#1475)
62a83fd 
<!--
Congratulations! You've made it this far! You're not quite done yet
though.

Once merged, your PR is going to appear in the release notes with the
title you set, so make sure it's a great title that fully reflects the
extent of your awesome contribution.

Then, please replace this with a description of the change and which
issue is fixed (if applicable). Please also include relevant motivation
and context. List any dependencies (if any) that are required for this
change.

Once you're done, someone will review your PR shortly (see the section
"Who can review?" below to tag some potential reviewers). They may
suggest changes to make the code even better. If no one reviewed your PR
after a week has passed, don't hesitate to post a new comment
@-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

- [ ] This PR fixes a typo or improves the docs (you can dismiss the
other checks if that's the case).
- [ ] Did you read the [contributor
guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the
[forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes?
Here are the
[documentation
guidelines](https://github.com/huggingface/transformers/tree/main/docs),
and
[here are tips on formatting
docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Anyone in the community is free to review the PR once the tests have
passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the
right person to tag with @

@OlivierDehaene OR @Narsil

 -->

---------

Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
Co-authored-by: Morgan Funtowicz <funtowiczmo@gmail.com>
Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@OlivierDehaene OlivierDehaene OlivierDehaene left review comments

@Narsil Narsil Awaiting requested review from Narsil

@mfuntowicz mfuntowicz Awaiting requested review from mfuntowicz

@echarlaix echarlaix Awaiting requested review from echarlaix

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@sywangyi @Narsil @yao-matrix @OlivierDehaene @mfuntowicz