Added tesnorizeation for avx2 based gemm. #3982
Conversation
|
Depends on this PR: #3981 |
There was a problem hiding this comment.
Thanks for the contribution.
I have left a couple of comments. But, I have a a high-level question, do we see performance improvement over FP32 using this.
Last time I checked FP32 VFMA is fully pipelined with a latency of 5 cycles. Int16 MADDs had a longer latency with little unclarity if they are pipelined. Basically Intel engineers spent lot of efforts for FP32 HW earlier.
| vec_b_1 = ins[1].vload([8, 0], "int8x32") | ||
| vec_one = tvm.const(1, "int16x16") | ||
| pair_reduction_0 = tvm.call_llvm_intrin('int16x16', | ||
| 'llvm.x86.avx2.pmadd.ub.sw', |
There was a problem hiding this comment.
Align (and other places as well)
| pair_reduction_1, vec_one) | ||
| if index == 0: | ||
| ib.emit(outs[0].vstore([0], quad_reduction_0)) | ||
| ib.emit(outs[0].vstore([8], quad_reduction_1)) |
There was a problem hiding this comment.
Why have 2 quad_reductions in assembly instead of unrolling in the schedule?
There was a problem hiding this comment.
Oh maybe, you would have then change num_lanes etc
If thats tha case, @yzhliu is working on making it an argument. Maybe we can use that to simplify this.
There was a problem hiding this comment.
@anijain2305. Correct. I wanted to be able to use the same memory layout which can be tensorized for avx2 and avx512.
|
@anijain2305, thats a good question. I havent checked the performance of this agains fp32 model we are interested in. I will report the numbers soon. But we do get a lot of speed up on skylake with avx512 and assuming we dont get anything from AVX2 instructions itself due to latency issues you mention, there should still be some perf left from the memory bw persepctive. |
Ok. This might be useful to do some paper calculations to see if we can get performance theoretically - https://www.agner.org/optimize/instruction_tables.pdf |
kimishpatel
force-pushed
the
tensorization_for_avx2
branch
2 times, most recently
from
5c61a2c to
841398d
Compare
|
@anijain2305, do you mind if I land this? I would like to use it for some of the stuff we are working on. |
| X = tvm.placeholder((m, k), name='X', dtype="uint8") | ||
| W = tvm.placeholder((n, k), name='W', dtype="int8") | ||
|
|
||
| #peak = 280 // This needs measurement and description of what this number is for avx2 machine. |
There was a problem hiding this comment.
Please remove the comments
Yes, please go ahead. The changes LGTM. Would appreciate if later you could update with the performance comparison. |
|
Thanks @anijain2305, will do so. |
|
Just as FYI. CI is failing on the test I added as it is dependent on this PR: #3981. I need to land that. |
Summary: Tensorized the same region as avx512. Names produce 16x1 int32 results. Does by doing two sets of AVX2 instructions to do reduction on 8x4 int8 kernel with 1x4 data. Test Plan: on avx2 machine: python tests/python/contrib/test_gemm_avx2_acc32.py Reviewers: Subscribers: Tasks: Tags:
Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:
kimishpatel
force-pushed
the
tensorization_for_avx2
branch
from
841398d to
c6bdd3c
Compare
|
@anijain2305, would mind merging this please? I dont think I can, plus it seems that my lint changes and addressing your comments might have voided your approval. |
|
I don't have merge permissions :) |
|
Thanks @kimishpatel @anijain2305 |
* Added tesnorizeation for avx2 based gemm. Summary: Tensorized the same region as avx512. Names produce 16x1 int32 results. Does by doing two sets of AVX2 instructions to do reduction on 8x4 int8 kernel with 1x4 data. Test Plan: on avx2 machine: python tests/python/contrib/test_gemm_avx2_acc32.py Reviewers: Subscribers: Tasks: Tags: * Fix lint errors. Removed commented out code. Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:
* Added tesnorizeation for avx2 based gemm. Summary: Tensorized the same region as avx512. Names produce 16x1 int32 results. Does by doing two sets of AVX2 instructions to do reduction on 8x4 int8 kernel with 1x4 data. Test Plan: on avx2 machine: python tests/python/contrib/test_gemm_avx2_acc32.py Reviewers: Subscribers: Tasks: Tags: * Fix lint errors. Removed commented out code. Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:
* Added tesnorizeation for avx2 based gemm. Summary: Tensorized the same region as avx512. Names produce 16x1 int32 results. Does by doing two sets of AVX2 instructions to do reduction on 8x4 int8 kernel with 1x4 data. Test Plan: on avx2 machine: python tests/python/contrib/test_gemm_avx2_acc32.py Reviewers: Subscribers: Tasks: Tags: * Fix lint errors. Removed commented out code. Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:
Summary:
Similar to other avx512 tensorization that reduces data:1x4 and kernel:16x4 to output:1x16, this PR introduces similar reduction using avx2 tensorization. It keeps the same API as avx512 so as to not have to introduce a new memory layout for weights.
Test Plan:
on avx2 machine:
python tests/python/contrib/test_gemm_avx2_acc32.py
Reviewers:
Subscribers:
Tasks:
Tags:
Thanks for contributing to TVM! Please refer to guideline https://docs.tvm.ai/contribute/ for useful information and tips. After the pull request is submitted, please request code reviews from Reviewers.