Fix SmoothQuant offload bug

[dsikka]

SUMMARY:

• Offload/onload properly during SmoothQuant calibration

We may actually not need this change?
The issue is because of the following line:

scales = activation_scales.pow(self.smoothing_strength) / weight_scales.pow(
            1 - self.smoothing_strength
        )

i.e y/x

However, we use y.div_(x) when actually applying the smoothing scales, which does it in place and does not cause an issue.

Not 100% sure why one is ok and one is not

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

[kylesayrs]

@dsikka Apparently pytorch does not throw errors for inplace operations on meta-device tensors.

a = torch.rand(10, device="meta")
# tensor(..., device='meta', size=(10,))

a.div_(6)
# no error
# tensor(..., device='meta', size=(10,))

Since the _apply_smoothing doesn't take this into account, it's likely that this function silently fails on offloaded weights

[dsikka]

@dsikka Apparently pytorch does not throw errors for inplace operations on meta-device tensors.

a = torch.rand(10, device="meta")
# tensor(..., device='meta', size=(10,))

a.div_(6)
# no error
# tensor(..., device='meta', size=(10,))

Since the _apply_smoothing doesn't take this into account, it's likely that this function silently fails on offloaded weights

Yeah same behaviour I saw. I'll update it for the actual case when smoothing is applied as well.

[rahul-tuli]

LGTM

[kylesayrs]

Grepped for .weight, lgtm!
Once accelerate utilities land, we can replace these with align_module_device