[Core] Change 8-bit serialization weight format format
#1164
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
@younesbelkada Thanks a lot for your work on this ❤️ I'm waiting with my final review and merge until you ping me, as you I remember you saying this is potentially still in flux. |
|
Thanks @Titus-von-Koeller ! yes let me run few other tests and ping you here |
bitsandbytes/nn/modules.py
Outdated
| if isinstance(weight_format, torch.Tensor): | ||
| weight_format = weight_format.item() | ||
|
|
||
| # For new weights format storage type we expclicitly check | ||
| # if weights_format is on the mapping | ||
| if isinstance(weight_format, int) and weight_format not in LINEAR_8BIT_WEIGHTS_FORMAT_MAPPING.values(): | ||
| raise ValueError(f"Expected supported weight format - got {weight_format}") | ||
| elif isinstance(weight_format, int) and weight_format in LINEAR_8BIT_WEIGHTS_FORMAT_MAPPING.values(): | ||
| weight_format = dict( | ||
| zip(LINEAR_8BIT_WEIGHTS_FORMAT_MAPPING.values(), LINEAR_8BIT_WEIGHTS_FORMAT_MAPPING.keys()) | ||
| )[weight_format] | ||
|
|
There was a problem hiding this comment.
622..633 looks like it should be a free function (determine_weight_format() or similar).
|
There should probably be at least some tests for this in this repo too. |
|
Thanks @akx for the review! Good catches. I agree that these changes would make sense. @younesbelkada In case you're too busy tmr, I can wrap this up for you and come up with some simple tests. Anyways, we can release this by Thursday, no problem, so that we're aligned with the Transformers release. For now, instead of this, I'll focus on some multi-backend related topics as these are critical to be well prepared for the meeting on that topic tmr evening. |
Co-authored-by: Aarni Koskela <akx@iki.fi>
Co-authored-by: Aarni Koskela <akx@iki.fi>
Co-authored-by: Aarni Koskela <akx@iki.fi>
…rs/bitsandbytes into fix-8bit-serialization
| # For new weights format storage type, we explicitly check | ||
| # if weights_format is on the mapping | ||
| if isinstance(weight_format, int) and weight_format not in INVERSE_LINEAR_8BIT_WEIGHTS_FORMAT_MAPPING: | ||
| raise ValueError(f"Expected supported weight format - got {weight_format}") | ||
| elif isinstance(weight_format, int) and weight_format in INVERSE_LINEAR_8BIT_WEIGHTS_FORMAT_MAPPING: | ||
| weight_format = INVERSE_LINEAR_8BIT_WEIGHTS_FORMAT_MAPPING[weight_format] |
There was a problem hiding this comment.
This doesn't seem to make much sense? Why is the isinstance() check repeated?
| if isinstance(weight_format, torch.Tensor): | ||
| weight_format = weight_format.item() | ||
|
|
||
| # For new weights format storage type, we explicitly check | ||
| # if weights_format is on the mapping | ||
| if isinstance(weight_format, int) and weight_format not in INVERSE_LINEAR_8BIT_WEIGHTS_FORMAT_MAPPING: | ||
| raise ValueError(f"Expected supported weight format - got {weight_format}") | ||
| elif isinstance(weight_format, int) and weight_format in INVERSE_LINEAR_8BIT_WEIGHTS_FORMAT_MAPPING: | ||
| weight_format = INVERSE_LINEAR_8BIT_WEIGHTS_FORMAT_MAPPING[weight_format] | ||
|
|
There was a problem hiding this comment.
As said before, this should probably be a free-standing helper function.
…s-foundation#1164) * change 8-bit serialization weight format format * precimmit * pre-commit * fix * Update bitsandbytes/nn/modules.py Co-authored-by: Aarni Koskela <akx@iki.fi> * Update bitsandbytes/nn/modules.py Co-authored-by: Aarni Koskela <akx@iki.fi> * Update bitsandbytes/utils.py Co-authored-by: Aarni Koskela <akx@iki.fi> * address feedback * lint --------- Co-authored-by: Aarni Koskela <akx@iki.fi>
…s-foundation#1164) * change 8-bit serialization weight format format * precimmit * pre-commit * fix * Update bitsandbytes/nn/modules.py Co-authored-by: Aarni Koskela <akx@iki.fi> * Update bitsandbytes/nn/modules.py Co-authored-by: Aarni Koskela <akx@iki.fi> * Update bitsandbytes/utils.py Co-authored-by: Aarni Koskela <akx@iki.fi> * address feedback * lint --------- Co-authored-by: Aarni Koskela <akx@iki.fi>
Currently for 8-bit layers the weight format are saved in pure str in the state dict, which is no longer supported in transformers
This PR should be totally backward compatible with previous 8-bit weights pushed on the Hub
cc @Titus-von-Koeller @TimDettmers @SunMarc