Multi-GPU setting in example gpt-neox-20b_peft

[Wsy002]

Hi, I have successfully completed all three steps in one GPU setting. I want to ask how to do multi-gpu fine tuning in the code below(Step 1)

https://github.com/lvwerra/trl/blob/main/examples/sentiment/scripts/gpt-neox-20b_peft/clm_finetune_peft_imdb.py

By the way I have searched some solutions and tried to modify some code based on #219 and peft PR #145 provided by @younesbelkada. But they seem
still not work. Could you please help with this.

One more question, I would like to ask whether all three steps(clm+ merge+PPO)support multi-gpu setting now? Thanks.

[younesbelkada]

hi @Wsy002

If I understood correctly you want to do https://github.com/lvwerra/trl/blob/main/examples/sentiment/scripts/gpt-neox-20b_peft/clm_finetune_peft_imdb.py in a multi-GPU setup
For that you need to use the main branch of transformers as it contains a fix for Trainer + multi-GPU: huggingface/transformers#22532
Can you try with that and let us know if it works?