Huge performance decrease by quantization

[kice]

I use the code from PR #13715, and I got a huge performance decrease by doing quantization on my model. I tested on Windows 10 with CUDA 10 and cudnn7 on Titan X (Pascal), using pre-release build from pip mxnet-cu100.

Alought by this issue #10897, it claimed that INT8 quantization can save GPU memory during usage, I got almose 2x more VRAM usage by quantization.

Do we excepted that INT8 quantization is super slow and use more memory on GPU?

And I may assume that UINT8 quantization is not yet supported since the UINT8 quantizated parameters is signed integer.

So, do we have any plan for improving INT8 quantization in the near future?

[marcoabreu]

@DickJC123

[pengzhao-intel]

FYI, https://github.com/apache/incubator-mxnet/blob/master/example/quantization/README.md

[kice]

@pengzhao-intel

Form your link

Performance is expected to decrease with GPU, however the memory footprint of a quantized model is smaller.

But in my case, it double the GPU memory usage. I don't think it can be considered as "smaller".

[pengzhao-intel]

FYI again, #13145 (comment)

[kice]

I couldn't find any information about why GPU memory usage increased.

[pengzhao-intel]

@ThomasDelteil do you have some data to show the memory changes of INT8 flow?

@reminisce to comment your question.

[vrakesh]

@mxnet-label-bot add [Operator, Performance]

[vrakesh]

@mxnet-label-bot add [Quantization]

[pengzhao-intel]

#9552 (comment)

Since there is no plan till now, closing this issue.
Welcome to file PR to fix related issues.