Master Build - AWQ - OOM when using GPU

[frenzybiscuit]

I am using the master build of llmcompressor from git clone.

When quantizing with AWQ 4bit with CPU only, it works fine, if slow.

When its set to "auto" it crashes from OOM errors.

Trying to use the following:

model = AutoModelForCausalLM.from_pretrained(
MODEL_ID, device_map="auto", torch_dtype="auto", max_memory={0: "23GIB", 1: "23GIB", "cpu": "90GIB"}

I'm sure the max_memory is wrong, im copying that from autoawq. I only added that because of the crashes.

Environment:

Python 3.11
Debian 12 64bit
2x3090
7950x

[brian-dellabetta]

Hi @frenzybiscuit , what is the size of the model you are using? You can try setting offload_device in AWQModifier to offload cached activations, but it will slow down your runtime

[frenzybiscuit]

I’m trying to do a 24B model. System has 128GB RAM and 2x3090.

AutoAWQ can do 70B quants on this hardware.

I will try that command when I’m home

[frenzybiscuit]

Hi @brian-dellabetta can you provide an example python script example that works for you on similar hardware?

I have tried to fiddle with the settings, but feel like I'm not doing them correctly.

I am planning on upgrading to 4x3090's in the near future, but since AutoAWQ can quant these on my hardware they should also be able to on llm-compressor.

[brian-dellabetta]

Hi @frenzybiscuit , I was able to run the example in #1444 on a single H100 device, though it required about 70GB of VRAM for 256 calibration samples. AutoAWQ takes some liberty with the kwargs passed into modules when calculating loss:

• kwargs are cached only for the first layer
• Those module_kwargs are passed in and injected into the layer configs
• They are used as kwargs for each self_attn layer even though they won't match the kwargs passed through in a vanilla forward pass of the model.

We have decided not to do this, instead caching the kwargs for each parent layer. So our forward passes are more true-to-form at the cost of higher memory requirements (which can be offloaded by setting offload_device=torch.device("cpu")). We are discussing internally a better alternative, I think we might be able to do this with hooks such that the smoothing is applied directly, rather than caching for several modules at a time and then applying smoothing.

[brian-dellabetta]

Just did a quick investigation with "Qwen/Qwen3-30B-A3B" (provided in #1444 example script). The model itself takes about 59GB in VRAM, and when caching kwargs it adds another 12GB or so. That is fairly marginal with respect to the size of the model, so the bigger fish to fry is sequentially onloading the model. It is quantized one layer at a time, but we currently onload all the layers at start-up. It looks like AutoAWQ has a lot of logic moving modules and kwargs onto best_device, we need to add this to our pipelining framework. That feature is a priority for us at the moment, you can follow that here:

• [Performance] Sequential onloading #1263

[frenzybiscuit]

Got it, so for the time being stick to runpod for quants with llm-compressor. Hopefully with the 4x3090 setup I'll be able to do 32B, although I'd really like to be able to do 70B quants with llm-compressor.