RLooTrainer bug when using deepspeed

[macheng6]

System Info

When using DeepSpeed, the RLOOTrainer reports an error: "ValueError: Please make sure to properly initialize your accelerator via accelerator = Accelerator() before using any functionality from the accelerate library." This is likely due to the accelerate not being properly initialized in line 120 of the RLOOTrainer code, possibly because the deepspeed_plugin was not passed in.

trl: 0.12.0.dev0
transformers: 4.45.2
accelerate: 1.0.1

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder
• My own task or dataset (give details below)

Reproduction

accelerate launch --config_file examples/accelerate_configs/deepspeed_zero3.yaml
examples/scripts/rloo/rloo.py
--dataset_name trl-internal-testing/descriptiveness-sentiment-trl-style
--dataset_train_split descriptiveness
--output_dir models/minimal/rloo
--rloo_k 2
--num_ppo_epochs 1
--num_mini_batches 1
--learning_rate 3e-6
--per_device_train_batch_size 1
--gradient_accumulation_steps 16
--total_episodes 10000
--model_name_or_path EleutherAI/pythia-1b-deduped
--sft_model_path EleutherAI/pythia-1b-deduped
--reward_model_path EleutherAI/pythia-1b-deduped
--local_rollout_forward_batch_size 1
--missing_eos_penalty 1.0

Expected behavior

This is likely due to the accelerate not being properly initialized in line 120 of the RLOOTrainer code, possibly because the deepspeed_plugin was not passed in.

[naskimed]

Hey, I have the same issue using PPOTrainer: "ValueError: Please make sure to properly initialize your accelerator via accelerator = Accelerator() before using any functionality from the accelerate library".

trl: 0.13.0.dev0
transformers: 4.46.2
accelerate: 1.1.0.dev0

[KAKSIS]

Hey, I have the same issue using PPOTrainer: "ValueError: Please make sure to properly initialize your accelerator via accelerator = Accelerator() before using any functionality from the accelerate library".

trl: 0.13.0.dev0 transformers: 4.46.2 accelerate: 1.1.0.dev0

I have the same problem

[kongjiellx]

+1
with PPOTrainer

[macheng6]

After using the version configuration below, the code can be run:
trl==0.11.4 accelerate==0.33.0，

[leobianco]

Having the same problem with RLOOTrainer.

[zwhe99]

+1

[yanan1116]

same error with /home/yanan/trl/examples/scripts/ppo/ppo.py.
accelerate: 1.2.0.dev0
trl: 0.13.0
transformers: 4.46.1

[yaswanthchittepu]

Same error with the ppo example script provided at the huggingface trl repo trl/examples/scripts/ppo/ppo_tldr.py, when using deepspeed zero2

acceelrate: 1.1.1
trl: 0.13.0.dev0
transformers: 4.47.0
deepspeed: 0.15.4

[karaszc]

Same error with the ppo example script at https://github.com/huggingface/trl/blob/main/examples/scripts/ppo/ppo.py, while using deepspeed zero3.

acceelrate: 1.1.1
trl: 0.15.2
transformers: 4.49.0
deepspeed: 0.15.4

Has anyone got this error cleared???

[kxzxvbk]

Downgrade accelerate==0.33.0 works for me.

[leobianco]

Still having this issue.

Previously I could downgrad accelerate to solve the problem, but now I really want to use newer versions of trl that apply the selective_log_softmax function, and they require higher versions of accelerate that have this bug.

So annoying.

[qgallouedec]

Should be now fixed with #3801, for PPO, it still needs heavy refactoring, so I would expect it not to work until then