[DataType] BF16 Support #17670
Merged
[DataType] BF16 Support #17670
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Hi @joshua-j-hong, thank you so much for the great contributions! Could you please add some PR descriptions so that CI tests can be triggered? |
Contributor
|
@tvm-bot rerun |
Contributor
|
Failed to re-run CI in https://github.com/apache/tvm/actions/runs/13505622336 Detailswith response |
Contributor
|
@tvm-bot rerun |
MasterJH5574
force-pushed
the
jjhong_bf16
branch
from
e2ee869 to
5491f0c
Compare
MasterJH5574
approved these changes
Mar 8, 2025
Contributor
MasterJH5574
left a comment
MasterJH5574
left a comment
There was a problem hiding this comment.
LGTM, thanks for rehabilitating BF16!
Contributor
@joshua-j-hong Just want to check these two points. Have we confirmed that they are in good shape now? We are good to merge if these are confirmed. |
Contributor
Author
|
I've added the changes for quantization in MLC LLM (will put up the corresponding PR soon) and am currently testing with the original model, but am running into some local build issues. Hoping to resolve these tomorrow to close out this ticket! |
MasterJH5574
pushed a commit
to mlc-ai/mlc-llm
that referenced
this pull request
Mar 10, 2025
Allows BF16 for model datatypes in group quantization and adds Quantization settings for BF16. Corresponding PR in TVM is apache/tvm#17670
ShiboXing
pushed a commit
to ShiboXing/tvm
that referenced
this pull request
Aug 10, 2025
Adds general BF16 support to TVM - Addresses missing legalization in `comm_reducer` - Adds BF16 legalization pass skipping if the target supports BF16 - Unit Tests for `comm_reducer` changes as well as legalization skipping - Modifications to TVM datatypes to allow for `T.bfloat16` in the test file - Fixes for BFloat related cuda codegen Related PR in MLC-LLM adds BF16 support with quantization mlc-ai/mlc-llm#3158 Tested with the original problematic model Gemma 2 27b with both added quantization configurations `q4bf16_0` and `q4bf16_1`. While compilation is successful and the first few rounds of prompting have expected performance, we observe that for long contexts generation quality degrades. The same behavior isn't observed on Gemma 2 9B, quantized or unquantized --------- Co-authored-by: Joshua Hong <jjhong@andrew.cmu.edu>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds general BF16 support to TVM
comm_reducercomm_reducerchanges as well as legalization skippingT.bfloat16in the test fileRelated PR in MLC-LLM adds BF16 support with quantization mlc-ai/mlc-llm#3158
Tested with the original problematic model Gemma 2 27b with both added quantization configurations
q4bf16_0andq4bf16_1. While compilation is successful and the first few rounds of prompting have expected performance, we observe that for long contexts generation quality degrades. The same behavior isn't observed on Gemma 2 9B, quantized or unquantized