Add Int4TilePackedTo4dTensor #2791
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/2791
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 22b937f with merge base 15a6de6 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
bot
added
the
CLA Signed
jerryzh168
changed the title
jerryzh168
force-pushed
the
int4wo-tinygemm
branch
from
0b18c16 to
e73434e
Compare
jerryzh168
added
the
topic: not user facing
17 tasks
jerryzh168
force-pushed
the
int4wo-tinygemm
branch
2 times, most recently
from
ed800a1 to
b878ee5
Compare
metascroy
reviewed
Aug 20, 2025
test/quantization/quantize_/workflows/int4/test_int4_tensor_core_tile_packed_tensor.py
Outdated
Show resolved
Hide resolved
metascroy
reviewed
Aug 20, 2025
metascroy
reviewed
Aug 20, 2025
torchao/quantization/quantize_/workflows/int4/int4_tensor_core_tile_packed_tensor.py
Outdated
Show resolved
Hide resolved
metascroy
reviewed
Aug 20, 2025
test/quantization/quantize_/workflows/int4/test_int4_tensor_core_tile_packed_tensor.py
Outdated
Show resolved
Hide resolved
jerryzh168
force-pushed
the
int4wo-tinygemm
branch
from
b878ee5 to
1922aaf
Compare
vkuzo
reviewed
Aug 20, 2025
torchao/quantization/quantize_/workflows/int4/int4_tensor_core_tile_packed_tensor.py
Outdated
Show resolved
Hide resolved
jerryzh168
force-pushed
the
int4wo-tinygemm
branch
7 times, most recently
from
1be7de0 to
4046cc0
Compare
jerryzh168
changed the title
jerryzh168
force-pushed
the
int4wo-tinygemm
branch
from
4046cc0 to
f0b97a6
Compare
vkuzo
reviewed
Aug 27, 2025
vkuzo
reviewed
Aug 27, 2025
| """ | ||
| tile_packed_to_4d is referring to the format used by tensor core tiled kernels for int4 quantization | ||
| """ | ||
| TILE_PACKED_TO_4D = "tile_packed_to_4d" |
There was a problem hiding this comment.
lgtm, but in a separate PR would be good to delete this enum since we determined that PLAIN is the only format which is reused, and all the others are tensor-specific
There was a problem hiding this comment.
we also want to delete global PackingFormat as well right?
vkuzo
reviewed
Aug 27, 2025
| ) | ||
|
|
||
| original_shape = hp_tensor.shape | ||
| # use a fixed value to simplify api |
There was a problem hiding this comment.
use fixed inner_k_tiles here to have a shorter arg list, I didn't see people change it anywhere
vkuzo
reviewed
Aug 27, 2025
torchao/quantization/quantize_/workflows/int4/int4_tile_packed_to_4d_tensor.py
Outdated
Show resolved
Hide resolved
jerryzh168
force-pushed
the
int4wo-tinygemm
branch
4 times, most recently
from
6411b23 to
b2885aa
Compare
metascroy
approved these changes
Aug 28, 2025
jerryzh168
force-pushed
the
int4wo-tinygemm
branch
from
b2885aa to
99bdf5d
Compare
… 4d packing This commit introduces Int4TilePackedTo4dTensor, a new tensor subclass for int4 weight-only quantization using tensor core tiled packing format. Key features: - Implements tensor core tiled packing for efficient computation on tensor cores - Supports PackingFormat.TILE_PACKED_TO_4D in Int4WeightOnlyConfig version 2 - Optimized for tinygemm int4mm kernel (_weight_int4pack_mm) - Includes comprehensive test suite The implementation follows the same pattern as other int4 tensor subclasses but uses a specialized packing format optimized for tensor core matrix multiplication performance. Changes: - Add Int4TilePackedTo4dTensor implementation - Update Int4WeightOnlyConfig version 2 to support TILE_PACKED_TO_4D packing format - Add TILE_PACKED_TO_4D to PackingFormat enum - Add comprehensive tests including serialization, different group sizes, and error conditions - Update __init__.py files to export new tensor class Test: python test/quantization/quantize_/workflows/int4/test_int4_tile_packed_to_4d_tensor.py
jerryzh168
force-pushed
the
int4wo-tinygemm
branch
from
99bdf5d to
22b937f
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit introduces Int4TilePackedTo4dTensor, a new tensor subclass for int4 weight-only quantization using tensor core tiled packing format.
Key features:
The implementation follows the same pattern as other int4 tensor subclasses but uses
a specialized packing format optimized for tensor core matrix multiplication performance.
Changes:
Test:
python test/quantization/quantize_/workflows/int4/test_int4_tile_packed_to_4d_tensor.py