[Question] Any way to access the FP32 checkpoints?

[veritas9872]

Hello!
I am currently doing research into quantizing Llama models.
However, I have noticed that the Llama checkpoints made public are always in BF16, including the embeddings.
Although this may be suitable for most use cases, I am currently trying some research that may be more precision-sensitive than most works.
Is there any way to access the master FP32 weights? It is my understanding that all weights have their master copies in FP32 but have some operations that are computed in BF16. I would be most grateful if the FP32 checkpoints could be made accessible.

[d-kleine]

Why do not just use Huggingface for that? You can specify the dtype as FP32 there.

[veritas9872]

The HuggingFace checkpoints are in BF16. Even when I load the results in FP32, they are simply BF16 checkpoints cast to FP32.