Surpport kv cache int8/int4 for triton backend

[yuguo-Jack]

Motivation

surpport kv cache int8/int4 for triton backend

Modifications

use c8 cmd:
--kv-cache-dtype int8
use c4 cmd:
--kv-cache-dtype int4 --kvint4-groupsize 32

[merrymercy]

Thanks for the contribution! They look good and i have a few comments.

[merrymercy]

Hi @yuguo-Jack Can you add an end-to-end accuracy test, similar to this one?

sglang/test/srt/test_triton_attn_backend.py

Line 31 in a5114b6

def test_mmlu(self):

[liangan1]

Except for the 'cuda', 'xpu' is also supported in the main branch. Change to use device=device as the original code? Also apply to other codes.

[yuguo-Jack]

ok

[merrymercy]

@yuguo-Jack Can you follow up on this? This is a high priority item and we would like to merge this as soon as possible once an accuracy unit test is added

[HaiShaw]

Does this feature only comes with compressed-tensors?
Can we decouple a bit, and add torchao's INT4/INT8 support too?

[yuguo-Jack]

Can reuse a8w8 linear in vllm

[merrymercy]

@yuguo-Jack Can you resolve the conflicts and add some correctness tests?

1. Kernel-level unit tests. Make sure to compare it against a reference implementation, which can be a pytorch implementation or a triton implementation in fp16.
2. End-to-end unit tests. Test the MMLU score.

   sglang/test/srt/test_triton_attention_backend.py

   Line 31 in 5e00dde

   def test_mmlu(self):

[merrymercy]

Let me know when the tests are added. I will review it again.