[REQUEST] Faster 6/8-bit EXL2 quantization

[grimulkan]

Problem

Any tips or suggestions on how we could generate a faster quantization, at the cost of accuracy, and still remain EXL2-compatible? Is a single pass possible, maybe at the cost of not exactly hitting the target avg. bit depth? Would skipping activation-aware quantization be worth it for 6 or 8 bit?

load-in-8-bit works reasonably on-the-fly, I'm looking for some way to re-create something like that, but runnable with Exllama.

One application is testing intermediate CPs while training. Would running measurement on the base model, then quantizing later checkpoints as we fine-tune without re-measuring perform reasonably, egs., at 6 or 8 bits to minimize the error?

Solution

Just a discussion is fine, maybe we don't need API changes.

Alternatives

No response

Explanation

EXL2 quantization of larger models can take a really long time.

Why EXL2 compatible? Exllama has wider compatibility with consumer hardware than other engines (egs., vLLM), and better efficiency than transformers, including TP support, and is faster than GGUF with GPUs.

Examples

No response

Additional context

No response

Acknowledgements

• I have looked for similar requests before submitting this one.
• I understand that the developers have lives and my issue will be answered when possible.
• I understand the developers of this program are human, and I will make my requests politely.