Can I fine tuning a llama model with pcw?

[naknak-choi]

Hi there! (if it's rude let me know!)

In the paper, "we believe that further training of an LLM with pcw could further enhance the abilities demonstrated in this work"
So, if I just add training code, LlamaForCausalLMPCW model can be trained??

thank you!

[nirrai21]

Hi,
The short answer is yes, though though I suppose you will need to modify the default transformer's training.

[naknak-choi]

Thanks for replying!

That's mean when I give training data to transformer with pcw, I should override LlamaForCausalLM forward on LlamaForCausalLMPCW according to pcw attention?

[naknak-choi]

Then if I don't want use Hugging face Trainer means implement a training code, I don't need to modify the default transformer's training right???

[nirrai21]

I am afraid I won't be able to assist you with the technical issues here since we didn't implement any training mechanism in our paper. Good luck.