add support for opimum bettertransformers

[winglian]

https://pytorch.org/blog/out-of-the-box-acceleration/

testing initial support for gpt-neox arch

[NanoCode012]

I checked that Pytorch link. This is not a torch 2.0 feature only right?

Edit: I also see a float16. This is different from fp16?

[winglian]

so I had to add float16 as an option b/c there is no way to have the model load as float16 currently without enabling automatic mixed precision which kicks in when you pass fp16 or bf16 to the trainer.

[winglian]

might be easiest to warn or raise a ValueError if they are using torch.__version__.split(".")[0] < 2 ?

[NanoCode012]

might be easiest to warn or raise a ValueError if they are using torch.__version__.split(".")[0] < 2 ?

I think this is good.

so I had to add float16 as an option b/c there is no way to have the model load as float16 currently without enabling automatic mixed precision which kicks in when you pass fp16 or bf16 to the trainer.

I just worry this will be confusing.. Could we just check for cfg.flash_optimum then handle it better instead of creating a new config?

[winglian]

what about cfg.flash_optimum_float32? this way we default to float16 when cfg.flash_optimum is set (the sane default), but if they really wanted float32, they can set it here.

[NanoCode012]

what about cfg.flash_optimum_float32? this way we default to float16 when cfg.flash_optimum is set (the sane default), but if they really wanted float32, they can set it here.

Hm, by default, I think people would expect their code to be float32. Would cfg.flash_optimum_float16 be better?

Edit: Could you also add it to Readme in case I forget to later?

[winglian]

Hm, by default, I think people would expect their code to be float32. Would cfg.flash_optimum_float16 be better?

so in doing some additional experiments, there are cases where I need to explicitly load in float16 where I'm not using bettertransformers

[NanoCode012]

I think some parts have been discussed before, but not sure if we decided on something.