Fix int4pack_mm error #517

yanbing-j · 2024-07-17T08:30:35Z

Need update meta shape in PyTorch first pytorch/pytorch#130915.

pytorch-bot · 2024-07-17T08:30:39Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/517

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 8aadb7d with merge base afde175 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

int4 tinygemm quantization is currently broken in master and being fixed in #517. Let's skip these tests for now until that is fixed.

manuelcandales · 2024-07-25T19:56:39Z

torchao/quantization/utils.py

@@ -349,6 +350,8 @@ def groupwise_affine_quantize_tensor_from_qparams(
 quant_max = 2 ** n_bit - 1

 int_data = quantize_affine(w, block_size, scales, zeros, output_dtype, quant_min, quant_max, zero_point_domain = ZeroPointDomain.FLOAT)
+ if TORCH_VERSION_AFTER_2_5:
+ int_data = (int_data[::, ::2] << 4 | int_data[::, 1::2]).to(torch.uint8)


This should break on MPS backend, since __lshift__.Scalar is not currently implemented for MPS

Is int_data in MPS device in this function? If so, we can make int_data in cpu device, then convert back to MPS device.

@malfet landed pytorch/pytorch#131813, so this won't be a problem anymore

In any case, I learned from @malfet today (see his suggestion on line 203) that if instead of using << in here, we use torch.bitwise_left_shift(x, 4), it would be falling back to cpu. So, things would work even prior to his PR having landed, if torch.bitwise_left_shift is used instead of <<

Thanks for the clarification. With pytorch/pytorch#131813, __lshift__.Scalar has MPS dispatch now.

msaroufim · 2024-07-25T20:28:24Z

@yanbing-j what's the status on this PR? If a breaking change requires more than 1 week of work to figure out on our end the right solution is to revert the offending PR

yanbing-j · 2024-07-26T01:28:13Z

@msaroufim This PR is pending on pytorch/pytorch#130915, which is blocked by the RuntimeError: CUDA error: invalid device function when using OpInfo.
After pytorch/pytorch#130915 is merged into PyTorch, current PR can fix int4 error in torchao.

yanbing-j · 2024-07-26T02:02:13Z

@msaroufim I update pytorch/pytorch#130915 not to use OpInfo.

malfet

[EDIT] Please ignore, both CUDA and MPS change will land at the same time

malfet · 2024-07-26T15:35:01Z

torchao/quantization/utils.py

@@ -349,6 +350,8 @@ def groupwise_affine_quantize_tensor_from_qparams(
 quant_max = 2 ** n_bit - 1

 int_data = quantize_affine(w, block_size, scales, zeros, output_dtype, quant_min, quant_max, zero_point_domain = ZeroPointDomain.FLOAT)
+ if TORCH_VERSION_AFTER_2_5:
+ int_data = (int_data[::, ::2] << 4 | int_data[::, 1::2]).to(torch.uint8)


Suggested change

int_data = (int_data[::, ::2] << 4 | int_data[::, 1::2]).to(torch.uint8)

int_data = (torch.bitwise_left_shift(int_data[::, ::2], 4) | int_data[::, 1::2]).to(torch.uint8)

malfet · 2024-07-26T15:35:17Z

torchao/prototype/hqq/hqq_tinygemm_linear.py

@@ -198,6 +199,8 @@ def hqq_quants_to_torch_quants(
 .reshape(shape)
 .contiguous()
 )
+ if TORCH_VERSION_AFTER_2_5:
+ W_q = (W_q[::, ::2] << 4 | W_q[::, 1::2]).to(torch.uint8)


Suggested change

W_q = (W_q[::, ::2] << 4 | W_q[::, 1::2]).to(torch.uint8)

W_q = (torch.bitwise_left_shift(W_q[::, ::2], 4) | W_q[::, 1::2]).to(torch.uint8)

msaroufim · 2024-07-28T18:42:56Z

Hi @yanbing-j just a heads up since I haven't seen CI be green, we're planning a release on Friday Aug 8 and doing a codefreeze on Friday Aug 2 so if this PR can't be landed in by this Wednesday I will have no choice but to revert your changes in core since this is a feature we have customers depend on such as https://github.com/mobiusml/hqq

yanbing-j · 2024-07-29T02:18:08Z

@msaroufim Thanks for the information. Could you please start this CI again? Thanks!

yanbing-j · 2024-07-29T02:47:06Z

@msaroufim @jerryzh168 I find pytorch/pytorch@6de65d5 will break test_int8_weight_only_quant_subclass and test_int4_weight_only_quant_subclass_api. Today's nightly can work, but tomorrow's will not.
Test plan:
python test/integration/test_integration.py -k test_int8_weight_only_quant_subclass_api
python test/integration/test_integration.py -k test_int4_weight_only_quant_subclass_api

msaroufim · 2024-07-29T16:40:07Z

Thanks @yanbing-j!

pytorch/pytorch@6de65d5 was reverted so indeed should only see breakages for 1 day

int4 tinygemm quantization is currently broken in master and being fixed in pytorch#517. Let's skip these tests for now until that is fixed.

* Fix int4pack_mm error * fix CI * Fix CI * Fix CI * Fix CI * Fix CI

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 17, 2024

svekars requested review from msaroufim, jerryzh168 and andrewor14 and removed request for andrewor14, msaroufim and jerryzh168 July 17, 2024 15:26

andrewor14 added a commit that referenced this pull request Jul 17, 2024

Skip int4 QAT tests for nightly for now

fa233ec

int4 tinygemm quantization is currently broken in master and being fixed in #517. Let's skip these tests for now until that is fixed.

andrewor14 mentioned this pull request Jul 17, 2024

Skip int4 QAT tests for nightly for now #521

Merged

andrewor14 added a commit that referenced this pull request Jul 17, 2024

Skip int4 QAT tests for nightly for now

3a75936

int4 tinygemm quantization is currently broken in master and being fixed in #517. Let's skip these tests for now until that is fixed.

andrewor14 added a commit that referenced this pull request Jul 17, 2024

Skip int4 QAT tests for nightly for now (#521)

ec95afd

int4 tinygemm quantization is currently broken in master and being fixed in #517. Let's skip these tests for now until that is fixed.

andrewor14 requested review from jerryzh168 and msaroufim July 18, 2024 15:11

yanbing-j force-pushed the yanbing/fix_int4_woq branch 2 times, most recently from 49b47a2 to a11e455 Compare July 19, 2024 03:48

andrewor14 requested a review from HDCharles July 19, 2024 14:25

This was referenced Jul 25, 2024

Adapt to _convert_weight_to_int4pack new behavior #541

Closed

update the input weight of _convert_weight_to_int4pack to [n][k / 2] uint8 pytorch/pytorch#129940

Closed

manuelcandales reviewed Jul 25, 2024

View reviewed changes

yanbing-j mentioned this pull request Jul 26, 2024

Fix meta error in _convert_weight_to_int4pack pytorch/pytorch#130915

Closed

malfet reviewed Jul 26, 2024

View reviewed changes

yanbing-j added 4 commits July 27, 2024 19:56

Fix int4pack_mm error

283fa73

fix CI

cb4a5e4

Fix CI

b30667e

Fix CI

5f41c1e

yanbing-j force-pushed the yanbing/fix_int4_woq branch from ecd2a86 to 5f41c1e Compare July 28, 2024 02:57

Fix CI

bd3b79a

Fix CI

8aadb7d

yanbing-j force-pushed the yanbing/fix_int4_woq branch from f03a014 to 8aadb7d Compare July 29, 2024 06:37

msaroufim approved these changes Jul 29, 2024

View reviewed changes

msaroufim merged commit 8fa11a6 into pytorch:main Jul 29, 2024
13 checks passed

dbyoung18 pushed a commit to dbyoung18/ao that referenced this pull request Jul 31, 2024

Skip int4 QAT tests for nightly for now (pytorch#521)

53562fe

int4 tinygemm quantization is currently broken in master and being fixed in pytorch#517. Let's skip these tests for now until that is fixed.

dbyoung18 pushed a commit to dbyoung18/ao that referenced this pull request Jul 31, 2024

Fix int4pack_mm error (pytorch#517)

4563492

* Fix int4pack_mm error * fix CI * Fix CI * Fix CI * Fix CI * Fix CI

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix int4pack_mm error #517

Fix int4pack_mm error #517

yanbing-j commented Jul 17, 2024

pytorch-bot bot commented Jul 17, 2024 •

edited

Loading

manuelcandales Jul 25, 2024 •

edited

Loading

yanbing-j Jul 26, 2024

manuelcandales Jul 26, 2024

manuelcandales Jul 26, 2024 •

edited

Loading

yanbing-j Jul 28, 2024

msaroufim commented Jul 25, 2024 •

edited

Loading

yanbing-j commented Jul 26, 2024

yanbing-j commented Jul 26, 2024

malfet left a comment •

edited

Loading

malfet Jul 26, 2024

malfet Jul 26, 2024

msaroufim commented Jul 28, 2024

yanbing-j commented Jul 29, 2024

yanbing-j commented Jul 29, 2024

msaroufim commented Jul 29, 2024

	int_data = (int_data[::, ::2] << 4 \| int_data[::, 1::2]).to(torch.uint8)
	int_data = (torch.bitwise_left_shift(int_data[::, ::2], 4) \| int_data[::, 1::2]).to(torch.uint8)

	W_q = (W_q[::, ::2] << 4 \| W_q[::, 1::2]).to(torch.uint8)
	W_q = (torch.bitwise_left_shift(W_q[::, ::2], 4) \| W_q[::, 1::2]).to(torch.uint8)

Fix int4pack_mm error #517

Fix int4pack_mm error #517

Conversation

yanbing-j commented Jul 17, 2024

pytorch-bot bot commented Jul 17, 2024 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/517

✅ No Failures

manuelcandales Jul 25, 2024 • edited Loading

Choose a reason for hiding this comment

yanbing-j Jul 26, 2024

Choose a reason for hiding this comment

manuelcandales Jul 26, 2024

Choose a reason for hiding this comment

manuelcandales Jul 26, 2024 • edited Loading

Choose a reason for hiding this comment

yanbing-j Jul 28, 2024

Choose a reason for hiding this comment

msaroufim commented Jul 25, 2024 • edited Loading

yanbing-j commented Jul 26, 2024

yanbing-j commented Jul 26, 2024

malfet left a comment • edited Loading

Choose a reason for hiding this comment

malfet Jul 26, 2024

Choose a reason for hiding this comment

malfet Jul 26, 2024

Choose a reason for hiding this comment

msaroufim commented Jul 28, 2024

yanbing-j commented Jul 29, 2024

yanbing-j commented Jul 29, 2024

msaroufim commented Jul 29, 2024

pytorch-bot bot commented Jul 17, 2024 •

edited

Loading

manuelcandales Jul 25, 2024 •

edited

Loading

manuelcandales Jul 26, 2024 •

edited

Loading

msaroufim commented Jul 25, 2024 •

edited

Loading

malfet left a comment •

edited

Loading