[UX] Allow quantization of weights without calibration data

[mgoin]

In the case of using round-to-nearest or static scaling for quantization formats like FP8 weights i.e. using QuantizationModifier on weights, there is no need for calibration data. Ideally, there should be no forward pass required at all.

Proposed UX:

from transformers import AutoModelForCausalLM
from compressed_tensors.quantization import QuantizationArgs, QuantizationType, QuantizationScheme, QuantizationStrategy
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.transformers import oneshot

model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m")

FP8_W8 = QuantizationScheme(
    targets=["Linear"],
    weights=QuantizationArgs(
        num_bits=8,
        type=QuantizationType.FLOAT,
        strategy=QuantizationStrategy.TENSOR,
        symmetric=True,
        dynamic=False,
    ),
)

recipe = QuantizationModifier(
    config_groups={"group_0": FP8_W8},
    ignore=["lm_head"],
)

oneshot(
    model=model,
    recipe=recipe,
)

[Satrat]

Hey @mgoin, we do have a PR up for this in compressed-tensors: neuralmagic/compressed-tensors#108 where the weight/scale just get initialized when quantization is applied. Still some more work to do here as it doesn't account for the per-token use case that also doesn't require calibration.

[mgoin]

Very nice, thanks @Satrat. Could you share an example of what the UX would be like? I'm not sure how quantization is applied outside of oneshot, or if it will run without the dataset argument.

[Satrat]

Update here, two PRs are up that enable this feature:

• Model Offloading Support neuralmagic/compressed-tensors#113
• Model Offloading Support Pt 2 #34

An example of the UX is here: https://github.com/vllm-project/llm-compressor/blob/sa/big_model_support/examples/big_model_offloading/big_model_fp8.py

[Satrat]

Closing this out as the PRs mentioned above got merged, this feature is now in main