-
Notifications
You must be signed in to change notification settings - Fork 30
initial commit on compressed-tensors quantization support for fp8 #1011
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
DescriptionStart with a short description of what the PR does and how this is a change from The rest of the description includes relevant details and context, examples:
If the change fixes a bug or a Github issue, please include a link, e.g.,: TestsPlease describe how you tested this change, and include any instructions and/or ChecklistBefore submitting this PR, please make sure:
|
a0462b4 to
72a458f
Compare
Signed-off-by: Han Qi <hanq@google.com>
72a458f to
c0bd708
Compare
tpu_inference/layers/vllm/quantization/compressed_tensors/compressed_tensors.py
Outdated
Show resolved
Hide resolved
tpu_inference/layers/vllm/quantization/compressed_tensors/compressed_tensors_moe.py
Outdated
Show resolved
Hide resolved
tpu_inference/layers/vllm/quantization/compressed_tensors/compressed_tensors_moe.py
Outdated
Show resolved
Hide resolved
tpu_inference/layers/vllm/quantization/compressed_tensors/compressed_tensors_moe.py
Outdated
Show resolved
Hide resolved
Signed-off-by: Han Qi <hanq@google.com>
kyuyeunk
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've noticed that the code doesn't have any unit test? Can you add one?
tpu_inference/layers/vllm/quantization/compressed_tensors/compressed_tensors_moe.py
Outdated
Show resolved
Hide resolved
tpu_inference/layers/vllm/quantization/compressed_tensors/compressed_tensors_moe.py
Show resolved
Hide resolved
tpu_inference/layers/vllm/quantization/compressed_tensors/compressed_tensors_moe.py
Show resolved
Hide resolved
tpu_inference/layers/vllm/quantization/compressed_tensors/compressed_tensors_moe.py
Show resolved
Hide resolved
added unit test, PTAL. Thanks! |
Signed-off-by: Han Qi <hanq@google.com>
08c4667 to
6c68136
Compare
Signed-off-by: Han Qi <hanq@google.com>
Description
This PR is to enable models with FusedMoE module (such as Qwen3-coder) to run with compressed-tensors quantization scheme.
NOTE:
currently because of old jax version, we need to change
this line /home/hanq_google_com/uv_venv/lib/python3.12/site-packages/jax/experimental/pallas/ops/tpu/megablox/common.py: 44
which is:
To:
Tests
runs and produces good result.