Using the teacher model

[JJongyn]

Thank you for your work.

I am trying to train llama-58m from your code.
But, the models used as teacher in your code are LLAMA-58M, LLAMA-360M and GPT2-705M.
Do I need to use all these 3 models to train one student?

[shawnricecake]

Thank you for your work.

I am trying to train llama-58m from your code. But, the models used as teacher in your code are LLAMA-58M, LLAMA-360M and GPT2-705M. Do I need to use all these 3 models to train one student?

You only need LLaMA-360M and GPT2-705M, you can pretrain both of them first.