[FEA] W4A8 gemm surpport.

[Ageliss]

It seems we had INT4.GEMM and INT8.GEMM, even FP8.GEMM, but actually activations in LLMs are hard to be compressed. We sometime use W4A8 in the gemm, however, we first uncompress the two-INT4 stored in one INT8 and carry out the INT8.GEMM. This is ugly and very slow due to the memory reading and writing for INT4 decompress to INT8 process.

[FEA]: I ask for the surpport for INT4 * INT8 and output INT32/8.

W-A-KV bits:

figure is citing from the LLM.QAT paper.

[hwu36]

several sources:

w4a8 is already supported on hopper. check 3.3 and 3.4 changelog

on ampere, #1190 adds int4 support (@alexsamardzic ). FastTransformer also has it which requires a special layout (#911 @rhenry-nv). there is also a paper (https://arxiv.org/abs/2311.09550) about a w4a8 product solution.

[Ageliss]

several sources:

w4a8 is already supported on hopper. check 3.3 and 3.4 changelog

on ampere, #1190 adds int4 support (@alexsamardzic ). FastTransformer also has it which requires a special layout (#911 @rhenry-nv). there is also a paper (https://arxiv.org/abs/2311.09550) about a w4a8 product solution.

Thanks very much for your reply.

[alexsamardzic]

I'm now working on this feature - #1413 is created.