This repository has been archived by the owner on Nov 17, 2023. It is now read-only.
[MXNET-1446] Quantization: intgemm matrix multiply wrappers #17559
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 29 commits
November 28, 2019 15:41
import mxnet as mx a = mx.nd.random_uniform(low=-1.0, high=1.0, shape=[5, 64]) b = mx.nd.random_uniform(low=-1.0, high=1.0, shape=[8, 64]) b_scale = 127.0 / mx.nd.contrib.intgemm_maxabsolute(b).asscalar() b_prepared = mx.nd.contrib.intgemm_prepareb(b, multiplier = b_scale) mx.nd.FullyConnected(a, b, num_hidden=8, no_bias=True, flatten=False) mx.nd.contrib.intgemm_fully_connected(a, b_prepared, out_float_multiplier=1.0/b_scale, num_hidden=8, no_bias=True, flatten=False)
…lack VNNI support yet. This reverts commit 947f911.
…izedTransposed. This will make it easier to store a consistent file on disk.
|
cc @leezu to review the build logic. |
szha
reviewed
Aug 20, 2020
|
Otherwise LGTM. I reviewed tests and and op implementation. |
leezu
reviewed
Aug 25, 2020
|
@leezu Ready? |
leezu
reviewed
Aug 28, 2020
added 6 commits
August 28, 2020 18:06
…m CMakeLists.txt This reverts commit a5a441e.
|
@mxnet-bot run ci [unix-gpu] |
|
Jenkins CI successfully triggered : [unix-gpu] |
|
Thank you @kpuatamazon |
4 tasks
samskalicky
pushed a commit
that referenced
this pull request
Sep 16, 2020
* cherry-pick intgemm from master, fix build * Fix test to conform to 1.x * Makefile supporting intgemm compilation * Stricter dependencies on git checkout of intgemm * Operators depend on mkldnn * Don't compile intgemm with gcc older than 5 * Fix intgemm test for windows on 1.x by not using pytest * Update intgemm to use template arguments for integer immediates * Try to fix clang3.6 * Ban gcc < 5 in cmake * Update intgemm with gcc 5.5 debug workaround
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This pull request adds wrappers to the intgemm matrix multiplication library: https://github.com/kpu/intgemm .
A performance comparison with DNNL aka MKL-DNN is at kpu/intgemm#59
The library targets thin matrix sizes seen in neural machine translation inference and was part of the top submission to the 2018 Workshop on Neural Generation and Translation efficiency task: https://neural.mt/papers/edinburgh/wnmt_marian_paper.pdf . The purpose of this issue is to add similar functionality to Sockeye: awslabs/sockeye#771 .
Quantized Sockeye performance is 2.95x as fast. One problem with the current MXQuantizeSymbol approach is that Sockeye does not have a static graph for everything.
intgemm uses a custom memory layout for the weight matrix to make more memory accesses consecutive, so there are operators to convert weights to that format. The idea is that weights are typically loaded once for inference.
On architectures without VNNI, intgemm uses saturating 16-bit accumulation. This avoids an expensive madd_epi16 instruction every multiply by exploiting the fact that most neural network parameters are near 0.
Because x86 only offers a unsigned * signed instruction and most people want signed * signed, there are two strategies one can take.
Both intgemm and DNNL implement strategy 1; intgemm also implements strategy 2.
Similar to DNNL, intgemm has runtime CPUID selection among backends for SSSE3, AVX2, AVX512BW, and AVX512VNNI.
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.
Changes
intgemm_prepare_dataandintgemm_prepare_weightoperators to convert operands from fp32intgemm_take_weightfor taking weights in intgemm's weight format, which is useful for vocabulary shortlists in Sockeye.intgemm_fully_connectedfor matrix multiplyComments
Backward compatible.
intgemm requires the inner dimension be a multiple of 64 for efficiency and alignment reasons. Currently the outputs must be a multiple of 8 but there is in-progress code in intgemm to remove that.