[AWQ] Insane memory requirement: over 900GB for 32B model

[mratsim]

I tried to quantized GLM-4-0414-32B: https://huggingface.co/THUDM/GLM-4-32B-0414

Recipe:

recipe = [
    AWQModifier(
        bits=4,
        symmetric=False,
        # Read input->output from https://github.com/huggingface/transformers/blob/v4.51.3/src/transformers/models/glm4/modeling_glm4.py
        # which are somewhat easier than vllm ones as it's all in a single file
        mappings=[
            AWQMapping("re:.*input_layernorm", ["re:.*q_proj", "re:.*k_proj", "re:.*v_proj"]),
            AWQMapping("re:.*v_proj", ["re:.*o_proj"]),
            AWQMapping("re:.*post_attention_layernorm", ["re:.*gate_up_proj"]),
            AWQMapping("re:.*gate_up_proj", ["re:.*down_proj"]),
        ]
    ),
    QuantizationModifier(
        ignore=ignore_layers,
        config_groups={
            "group_0": QuantizationScheme(
                targets=["Linear"],
                weights=QuantizationArgs(
                    num_bits=4,
                    type=QuantizationType.INT,
                    dynamic=False,
                    symmetric=False,
                    strategy=QuantizationStrategy.GROUP,
                    group_size=128,
                ),
            ),
        },
    )
]

I tried using 128 samples as suggested in those slides ("Calibration set"): https://minjiazhang.github.io/courses/fall24-resource/slides/awq.pdf

However every sample memory usage grew by 1~5 GB leading in the end to over 900GB before I decided to give up on AWQ. Even with a swapfile, the time was spent in kernel swap in/out and IO and compute was slow to then frustratingly crash on Cuda OOM once that CPU part was solved.

Screenshot:

Side-note: couldn't the calibration be made multi-threaded?

[brian-dellabetta]

Hi @mratsim , thanks for taking an interest in our AWQ feature. Our initial port of AutoAWQ does not cache calibrated activations layer-by-layer, instead it does it for the entire model all at once. So that will increase host memory requirements substantially. Our experiments so far have been to validate the results AWQ published, nothing larger than an 8B-parameter model.

Once our work to extract the pipelining out into an abstraction that can be plugged into AWQ, a lot of your issues will be resolved -- #1279

There was some miscommunication about the current state of AWQ in llm-compressor. There are a few features we need to add in order for it to be really ready for non-experimental usage. We will make a new release and a broader announcement when that happens, sorry for the confusion.

Related to #1369

[frenzybiscuit]

@brian-dellabetta

I see #1279 was merged and this was marked as completed. However, I just tried to create a AWQ of a 24B model and watched the entire process kill itself due to OOM. Now, I only have 128GB RAM and 48GB VRAM for quantizing. I do not mind renting a server from runpod to create the AWQ, but I have no idea what the system requirements are for AWQ.

[brian-dellabetta]

Hi @frenzybiscuit , in my attempts I was able to run AWQ with a 30B model on a 80GB H100. IIRC it required about 65GB VRAM. System requirements are going to be dependent on model architecture and size as well as number of calibration samples used. I am not sure of any easy way to calculate this. But we do have a couple more PRs in flight to reduce memory requirements of llm-compressor and runtime performance of AWQ:

#1263
#1419
#1444

We plan to make a release soon with these changes.

[frenzybiscuit]

Thanks, Artus managed to do it on AutoAWQ which used around 180GB VRAM for a 70B model. At least I know how much compute to get through runpod now.

Looking forward to llm-compressors release.