Is there example of FP8 train LLM, pre-train or fine-tune

[harborn]

Feature request

I see the release version 1.12 has supported fp8, but I didn't see any example code for how to train LLM by using FP8.
How can I use FP8 to train model?

Motivation

I want to train model by using FP8.

Your contribution

NO.

[regisss]

v1.12 of Optimum Habana will actually be released in a few days.
Fp8 training is in a very early stage at the moment, that's why we have not added an example yet. It will get much better support in the following weeks and months!

For now, if you want to do it, you can check how it is done in this test that is part of our CI: https://github.com/huggingface/optimum-habana/blob/main/tests/test_fp8_examples.py