Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI for add gptq and awq int4 support in intel platform #2494

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

ErikKaum
Copy link
Member

@ErikKaum ErikKaum commented Sep 5, 2024

Run CI for pr #2444

Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
@sywangyi
Copy link
Contributor

sywangyi commented Sep 6, 2024

I check the 3 failure cases and find they are related with the bug fix in https://github.com/huggingface/text-generation-inference/pull/2444/files#diff-d8aff332cf9104dd7460d2f53575239dc1f4bcdd374e575b8a504568bfc2e078R325.
which will cause "Narsil/starcoder-gptq" 2 TP not to use exllama kernel. if you check 1TP of this model, which is using exllama kernel.
the generation result is close to the current 2TP result which is using exllama.

@sywangyi
Copy link
Contributor

sywangyi commented Sep 6, 2024

@Narsil to comment.

Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
@sywangyi
Copy link
Contributor

seems the failure is not related with the PR
ERROR integration-tests/models/test_flash_medusa.py::test_flash_medusa_simple - RuntimeError: Launcher crashed
ERROR integration-tests/models/test_flash_medusa.py::test_flash_medusa_all_params - RuntimeError: Launcher crashed
ERROR integration-tests/models/test_flash_medusa.py::test_flash_medusa_load - RuntimeError: Launcher crashed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants