Namespace Error when trying to train reward model

[bhauptvogel]

I ran python trainer_rm.py --configs defaults_rm oasst-rm-1-pythia-1.4b inside the model/model_training folder.

I got an error trying to train the model:

python trainer_rm.py --configs defaults_rm oasst-rm-1-pythia-1.4b
[2023-09-17 11:23:03,502] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
trainig_conf = Namespace(rng_seed=2703368087, is_reward_model=True, pooling='last', learning_rate='8e-6', gradient_checkpointing=False, gradient_accumulation_steps=4, per_device_train_batch_size=1, per_device_eval_batch_size=5, adam_beta1=0.9, adam_beta2=0.95, adam_epsilon='1e-12', weight_decay=0.0, warmup_steps=50, eval_steps=500, save_steps=1000, save_strategy='steps', max_length=2048, num_train_epochs=2, logging_steps=10, max_grad_norm=2.0, save_total_limit=4, dtype='float32', eval_accumulation_steps=None, freeze_layer=None, cache_dir='.cache', loss_fn='RMLoss', score_l2_reg=0.001, eval_size=None, log_dir='base', quantization=False, seq2seqmodel=False, fuse_gelu=True, log_wandb=True, verbose=False, output_dir='.saved_models_rm', use_custom_sampler=True, residual_dropout=0.01, use_flash_attention=True, sort_by_length=False, per_digit_tokens=False, datasets_extra=[], metrics=['accuracy', 'kendalltau'], deepspeed_config='configs/zero_config.json', max_replies=5, datasets=[{'oasst_export': {'lang': 'en,es,de,fr', 'hf_dataset_name': 'OpenAssistant/oasst1', 'val_split': 0.1}}, {'augment_oasst': {'input_file_path': 'augmented_latin_cyrillic_oasst_2023-03-27.jsonl'}}, {'anthropic_rlhf': {'fraction': 0.1, 'max_val_set': 1000}}, {'shp': {'max_val_set': 1000}}, {'hellaswag': {'fraction': 0.5, 'max_val_set': 1000}}, {'webgpt': {'val_split': 0.05, 'max_val_set': 1000}}, {'hf_summary_pairs': {'fraction': 0.1, 'max_val_set': 250}}], model_name='andreaskoepf/pythia-1.4b-gpt4all-pretrain', wandb_entity='open-assistant', local_rank=-1, deepspeed=False, resume_from_checkpoint=False, show_dataset_stats=False, world_size=1)
RNG seed: 2703368087
You are using a model of type gpt_neox to instantiate a model of type gpt_neox_reward_model. This is not supported for all configurations of models and can yield errors.
Some weights of GPTNeoXRewardModel were not initialized from the model checkpoint at andreaskoepf/pythia-1.4b-gpt4all-pretrain and are newly initialized: ['out_proj.bias', 'out_proj.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Number of trainable parameters: 1311M
Traceback (most recent call last):
  File "/home/benmainbird/Projects/ml/Open-Assistant/model/model_training/trainer_rm.py", line 334, in <module>
    main()
  File "/home/benmainbird/Projects/ml/Open-Assistant/model/model_training/trainer_rm.py", line 191, in main
    model = get_model(training_conf, tokenizer)
  File "/home/benmainbird/miniconda3/envs/oa/lib/python3.10/site-packages/model_training/utils/utils.py", line 353, in get_model
    residual_dropout_lima=conf.residual_dropout_lima,
AttributeError: 'Namespace' object has no attribute 'residual_dropout_lima'. Did you mean: 'residual_dropout'?

It worked (I didn't get the error when I added residual_dropout_lima: true to the config.yaml. (But I don't know if it is supposed to be true or what it does in general)
The following attributes are missing from the Namespace:

• residual_dropout_lima
• use_system_tag
• system_property_dropout
• system_add_length

I only tried the oasst-rm-1-pythia-1.4b model, the other model sizes have to be tested.

[RylanSchaeffer]

@bhauptvogel did you find a solution?