More fixes needed when QuantizationConfig is None #184
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dsikka
requested changes
Oct 7, 2024
| @@ -104,6 +104,8 @@ def from_pretrained( | |||
| """ | |||
| config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs) | |||
| compression_config = getattr(config, COMPRESSION_CONFIG_NAME, None) | |||
| if compression_config is None: | |||
| compression_config = getattr(config, QUANTIZATION_CONFIG_NAME, None) | |||
There was a problem hiding this comment.
Not required. See pathway used by HFQuantizer:
https://github.com/huggingface/transformers/blob/55be7c4c483a01a7e03e55a8756fc4385ec08ffc/src/transformers/quantizers/quantizer_compressed_tensors.py#L40
rahul-tuli
force-pushed
the
more-bugfixes
branch
from
a6dfdcc to
ecc49fc
Compare
kylesayrs
reviewed
Oct 7, 2024
dsikka
reviewed
Oct 7, 2024
| """ | ||
| if config.kv_cache_scheme is not None: | ||
| if config is not None and config.kv_cache_scheme is not None: |
There was a problem hiding this comment.
Do we use this function outside of apply_quantization_config? If not, we would never hit the None case?
kylesayrs
reviewed
Oct 7, 2024
| @@ -189,14 +190,14 @@ def apply_quantization_config( | |||
| return names_to_scheme | |||
|
|
|||
|
|
|||
| def process_quantization_config(config: QuantizationConfig) -> QuantizationConfig: | |||
| def process_quantization_config(config: Optional[QuantizationConfig]) -> Optional[QuantizationConfig]: | |||
There was a problem hiding this comment.
Seems like the only use of process_quantization_config is by apply_quantization_config.
What is the purpose of this change?
|
This was relevant to the reloading a compressed model case, as per discussion offline, this is not needed |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description taken from #180
CompressedTensorsHfQuantizer
attempts to useapply_quantization_config` to apply the quantization config to the modelHowever,
self.compressor.quantization_configcan beNonein the case that only sparsity is present. This does not align with the function contract ofapply_quantization_config. In such a case, an error is thrown.This PR builds on top of #180 and adds more bugfixes needed for the case when a None quantization config is passed in by the
HfQuantizer