GKD with Lora and fsdp?

[nivibilla]

Hi

I'm looking to distill a model from 70B into 500M, 1.5B and 3B for testing. However this won't fit into a single GPU so is it possible to use fsdp in combination with GKD and also can the student model be trained with Lora

Thanks

[kashif]

I believe that should be possible yes