[Float8] Make Inference and Training code independent

[drisspg]

Stacked PRs:

• [StaticQuant] add a linear observer class and test #807
• ->[Float8] Make Inference and Training code independent #808

---

Summary

Remove Old FP8 inference flow and and switch over to Float8MMConfig
in favor of the newly added support to AQT + quantize_ apis. It also completely separates and utilization on training code by creating a new Float8MMConfig and a addmm wrapper in the inference.py file

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/808

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit c5c333e with merge base f5703b0 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[vkuzo]

can we make the names ScaledMMConfig and Float8MMConfig not be confusing? Ideally it should be clear why there are two objects and in which context the user should use which object

[drisspg]

Sure, I was going to do ScaledMMConfigInference but that is even too verbose for me, any suggestions?

[vkuzo]

for training I went with a user facing Float8GemmConfig which is a dataclass and easy to explain (IMO), the way the data gets passed around is not user facing so details of ScaledMMConfig don't matter as much

tbh making ScaledMMConfig public and reusable between training + inference sounds right to me, if you really want this to be public without a dataclass wrapper

[drisspg]

It can not be a dataclass today because that wont work with compile, Lazos has a PR to make frozen dataclasses proxyable but until then it has to be a named tuple.

I love dataclasses but I dont understand why they are more understandable then a namedtuple, Is the problem that this name is similar to other names?

[vkuzo]

I think it's less about dataclass versus named tuple, and more about the field readability / understandability / future proofness, etc. For example, if we made ScaledMMConfig have an output dtype enum instead of a boolean and ensured all the other args are consistent with other public APIs, I think that would be fine.

https://stackoverflow.com/a/18348004/1058521 is one minor reason, default values look nicer with dataclasses