Quantized KV cache: update quanto

[zucchini-nlp]

What does this PR do?

The latest version of quanto was apparently refactored affecting quant-cache. This PR updates QuantoQuantizedCache to work with the latest version of quanto

[younesbelkada]

Thanks !
My only question being if these changes are Backward compatible, would this work with the previous quanto version? If not we could raise an error when initializing the QuantoCache educating users to install quanto>=0.2.0 , what do you think?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[zucchini-nlp]

@younesbelkada right, it's not compatible with older versions. I can add a warning or raise an error asking to update quanto to the latest version, if that's usual workflow in these situations? :)

[younesbelkada]

Since the quanto cache is not part of a release yet, I think we should just force users to use quanto >= 0.2.0 - for that I would just check the quanto version using something like this: https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py#L1576C53-L1576C79 and raise an error at the cache init if the quanto minimum version is not met
With respect to the imports we'll need to make sure to import AffineQuantizer and MaxOptimizer only if quanto version is greater than 0.2.0

[SunMarc]

LGTM ! I think it will be interesting to rerun the benchmark with quanto for int4 kv-cache quantization. @dacorvo said that the latency of models quantized with qint4 weights is drastically reduced (almost divided by two).

[younesbelkada]

Clean work, thanks !

[dacorvo]

LGTM ! I think it will be interesting to rerun the benchmark with quanto for int4 kv-cache quantization. @dacorvo said that the latency of models quantized with qint4 weights is drastically reduced (almost divided by two).

The new kernels only work when doing a matmul between fp16 inputs and int4 weights. In this case the KV are just dequantized, and I haven't bound the kernel for that (the CUDA dequantize method is included in quanto though so maybe you can give it a try).

[dacorvo]

LGTM ! I think it will be interesting to rerun the benchmark with quanto for int4 kv-cache quantization. @dacorvo said that the latency of models quantized with qint4 weights is drastically reduced (almost divided by two).

The new kernels only work when doing a matmul between fp16 inputs and int4 weights. In this case the KV are just dequantized, and I haven't bound the kernel for that (the CUDA dequantize method is included in quanto though so maybe you can give it a try).

More specifically: https://github.com/huggingface/quanto/blob/f545e01443767b8920609bfcde5417fe064eedfd/quanto/library/ext/cuda/awq/dequantize.cuh#L14
This is the fast int4 -> fp16 dequantizer for packed int4 "a la AWQ". This should be bound in pybind and called here: https://github.com/huggingface/quanto/blob/f545e01443767b8920609bfcde5417fe064eedfd/quanto/tensor/qbits/awq/qbits.py#L32

[zucchini-nlp]

@dacorvo i see, thanks! I'll check it out!

[amyeroberts]

Thanks for updating and adding the version guard!