RuntimeError: Unexpected key(s) in state_dict when loading OFAModel

[aaaaaannie]

Hi,I am encountering an issue when trying to load a pre-trained OFAModel. The error message I receive is as follows:

Environment Details:
pip version: 21.2.4
Fairseq version: Installed from the OFA repository

Steps to Reproduce:
Installed Fairseq from the OFA repository.
Configured the environment and downloaded the necessary pre-trained datasets.
Attempted to load the OFAModel using the provided scripts.

[taokz]

Could you let me know which scripts and checkpoints you are using?

[aaaaaannie]

Could you let me know which scripts and checkpoints you are using?

For the scripts, I used 'pretrain_tiny.sh' located in the scripts/pretrain/ . I only made two modifications to this script:

1. Changed CUDA_VISIBLE_DEVICES to 0,1,2,3
2. Set GPUS_PER_NODE to 4

For the checkpoint, I used 'biomedgpt_tiny.pt' which I downloaded from the Dropbox link provided in the checkpoints.md (https://www.dropbox.com/sh/cu2r5zkj2r0e6zu/AADZ-KHn-emsICawm9CM4MqVa?dl=0).

These were the key components I utilized for my setup. Let me know if you need any clarification or have additional questions about the configuration.

[taokz]

Could you try installing Fairseq from this repository instead of OFA and re-run the code? Additionally, could you please share the entire error log?

[aaaaaannie]

hi i've install fairseq from your repository, but still get the same error as below:

2024-09-23 02:08:07 - train.py[line:154] - INFO: training on 4 devices (GPUs/TPUs)
2024-09-23 02:08:07 - train.py[line:160] - INFO: max tokens per device = None and max sentences per device = 16
2024-09-23 02:08:07 - trainer.py[line:458] - INFO: Preparing to load checkpoint ../../scripts/biomedgpt_tiny.pt
Traceback (most recent call last):
File "/mypath/trainer.py", line 519, in load_checkpoint
Traceback (most recent call last):
File "/mypath/trainer.py", line 519, in load_checkpoint
state["model"], strict=True, model_cfg=self.cfg.model state["model"], strict=True, model_cfg=self.cfg.model

File "/mypath/fairseq/fairseq/distributed/module_proxy_wrapper.py", line 52, in load_state_dict
File "/mypath/fairseq/fairseq/distributed/module_proxy_wrapper.py", line 52, in load_state_dict
return self.module.module.load_state_dict(*args, **kwargs)
File "/mypath/fairseq/fairseq/models/fairseq_model.py", line 125, in load_state_dict
return self.module.module.load_state_dict(*args, **kwargs)
File "/mypath/fairseq/fairseq/models/fairseq_model.py", line 125, in load_state_dict
Traceback (most recent call last):
File "/mypath/trainer.py", line 519, in load_checkpoint
state["model"], strict=True, model_cfg=self.cfg.model
File "/mypath/fairseq/fairseq/distributed/module_proxy_wrapper.py", line 52, in load_state_dict
return self.module.module.load_state_dict(*args, **kwargs)
File "/mypath/fairseq/fairseq/models/fairseq_model.py", line 125, in load_state_dict
Traceback (most recent call last):
File "/mypath/trainer.py", line 519, in load_checkpoint
state["model"], strict=True, model_cfg=self.cfg.model
File "/mypath/fairseq/fairseq/distributed/module_proxy_wrapper.py", line 52, in load_state_dict
return self.module.module.load_state_dict(*args, **kwargs)
File "/mypath/fairseq/fairseq/models/fairseq_model.py", line 125, in load_state_dict
return super().load_state_dict(new_state_dict, strict)return super().load_state_dict(new_state_dict, strict)

  File "/root/miniconda3/envs/biomedgpt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1668, in load_state_dict

return super().load_state_dict(new_state_dict, strict) File "/root/miniconda3/envs/biomedgpt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1668, in load_state_dict
return super().load_state_dict(new_state_dict, strict)
File "/root/miniconda3/envs/biomedgpt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1668, in load_state_dict

File "/root/miniconda3/envs/biomedgpt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1668, in load_state_dict
self.class.name, "\n\t".join(error_msgs)))
self.class.name, "\n\t".join(error_msgs)))
RuntimeErrorRuntimeError: Error(s) in loading state_dict for OFAModel:
Unexpected key(s) in state_dict: "encoder.layers.0.attn_ln.weight", "encoder.layers.0.attn_ln.bias", "encoder.layers.0.ffn_layernorm.weight", "encoder.layers.0.ffn_layernorm.bias", "encoder.layers.0.self_attn.c_attn", "encoder.layers.1.attn_ln.weight", "encoder.layers.1.attn_ln.bias", "encoder.layers.1.ffn_layernorm.weight", "encoder.layers.1.ffn_layernorm.bias", "encoder.layers.1.self_attn.c_attn", "encoder.layers.2.attn_ln.weight", "encoder.layers.2.attn_ln.bias", "encoder.layers.2.ffn_layernorm.weight", "encoder.layers.2.ffn_layernorm.bias", "encoder.layers.2.self_attn.c_attn", "encoder.layers.3.attn_ln.weight", "encoder.layers.3.attn_ln.bias", "encoder.layers.3.ffn_layernorm.weight", "encoder.layers.3.ffn_layernorm.bias", "encoder.layers.3.self_attn.c_attn", "decoder.layers.0.self_attn_ln.weight", "decoder.layers.0.self_attn_ln.bias", "decoder.layers.0.cross_attn_ln.weight", "decoder.layers.0.cross_attn_ln.bias", "decoder.layers.0.ffn_layernorm.weight", "decoder.layers.0.ffn_layernorm.bias", "decoder.layers.0.self_attn.c_attn", "decoder.layers.0.encoder_attn.c_attn", "decoder.layers.1.self_attn_ln.weight", "decoder.layers.1.self_attn_ln.bias", "decoder.layers.1.cross_attn_ln.weight", "decoder.layers.1.cross_attn_ln.bias", "decoder.layers.1.ffn_layernorm.weight", "decoder.layers.1.ffn_layernorm.bias", "decoder.layers.1.self_attn.c_attn", "decoder.layers.1.encoder_attn.c_attn", "decoder.layers.2.self_attn_ln.weight", "decoder.layers.2.self_attn_ln.bias", "decoder.layers.2.cross_attn_ln.weight", "decoder.layers.2.cross_attn_ln.bias", "decoder.layers.2.ffn_layernorm.weight", "decoder.layers.2.ffn_layernorm.bias", "decoder.layers.2.self_attn.c_attn", "decoder.layers.2.encoder_attn.c_attn", "decoder.layers.3.self_attn_ln.weight", "decoder.layers.3.self_attn_ln.bias", "decoder.layers.3.cross_attn_ln.weight", "decoder.layers.3.cross_attn_ln.bias", "decoder.layers.3.ffn_layernorm.weight", "decoder.layers.3.ffn_layernorm.bias", "decoder.layers.3.self_attn.c_attn", "decoder.layers.3.encoder_attn.c_attn". :

During handling of the above exception, another exception occurred:

Error(s) in loading state_dict for OFAModel:
Unexpected key(s) in state_dict: "encoder.layers.0.attn_ln.weight", "encoder.layers.0.attn_ln.bias", "encoder.layers.0.ffn_layernorm.weight", "encoder.layers.0.ffn_layernorm.bias", "encoder.layers.0.self_attn.c_attn", "encoder.layers.1.attn_ln.weight", "encoder.layers.1.attn_ln.bias", "encoder.layers.1.ffn_layernorm.weight", "encoder.layers.1.ffn_layernorm.bias", "encoder.layers.1.self_attn.c_attn", "encoder.layers.2.attn_ln.weight", "encoder.layers.2.attn_ln.bias", "encoder.layers.2.ffn_layernorm.weight", "encoder.layers.2.ffn_layernorm.bias", "encoder.layers.2.self_attn.c_attn", "encoder.layers.3.attn_ln.weight", "encoder.layers.3.attn_ln.bias", "encoder.layers.3.ffn_layernorm.weight", "encoder.layers.3.ffn_layernorm.bias", "encoder.layers.3.self_attn.c_attn", "decoder.layers.0.self_attn_ln.weight", "decoder.layers.0.self_attn_ln.bias", "decoder.layers.0.cross_attn_ln.weight", "decoder.layers.0.cross_attn_ln.bias", "decoder.layers.0.ffn_layernorm.weight", "decoder.layers.0.ffn_layernorm.bias", "decoder.layers.0.self_attn.c_attn", "decoder.layers.0.encoder_attn.c_attn", "decoder.layers.1.self_attn_ln.weight", "decoder.layers.1.self_attn_ln.bias", "decoder.layers.1.cross_attn_ln.weight", "decoder.layers.1.cross_attn_ln.bias", "decoder.layers.1.ffn_layernorm.weight", "decoder.layers.1.ffn_layernorm.bias", "decoder.layers.1.self_attn.c_attn", "decoder.layers.1.encoder_attn.c_attn", "decoder.layers.2.self_attn_ln.weight", "decoder.layers.2.self_attn_ln.bias", "decoder.layers.2.cross_attn_ln.weight", "decoder.layers.2.cross_attn_ln.bias", "decoder.layers.2.ffn_layernorm.weight", "decoder.layers.2.ffn_layernorm.bias", "decoder.layers.2.self_attn.c_attn", "decoder.layers.2.encoder_attn.c_attn", "decoder.layers.3.self_attn_ln.weight", "decoder.layers.3.self_attn_ln.bias", "decoder.layers.3.cross_attn_ln.weight", "decoder.layers.3.cross_attn_ln.bias", "decoder.layers.3.ffn_layernorm.weight", "decoder.layers.3.ffn_layernorm.bias", "decoder.layers.3.self_attn.c_attn", "decoder.layers.3.encoder_attn.c_attn".

During handling of the above exception, another exception occurred:

self.__class__.__name__, "\n\t".join(error_msgs)))Traceback (most recent call last):

self.class.name, "\n\t".join(error_msgs))) File "../../train.py", line 537, in

Traceback (most recent call last):
RuntimeError File "../../train.py", line 537, in
RuntimeError: Error(s) in loading state_dict for OFAModel:
Unexpected key(s) in state_dict: "encoder.layers.0.attn_ln.weight", "encoder.layers.0.attn_ln.bias", "encoder.layers.0.ffn_layernorm.weight", "encoder.layers.0.ffn_layernorm.bias", "encoder.layers.0.self_attn.c_attn", "encoder.layers.1.attn_ln.weight", "encoder.layers.1.attn_ln.bias", "encoder.layers.1.ffn_layernorm.weight", "encoder.layers.1.ffn_layernorm.bias", "encoder.layers.1.self_attn.c_attn", "encoder.layers.2.attn_ln.weight", "encoder.layers.2.attn_ln.bias", "encoder.layers.2.ffn_layernorm.weight", "encoder.layers.2.ffn_layernorm.bias", "encoder.layers.2.self_attn.c_attn", "encoder.layers.3.attn_ln.weight", "encoder.layers.3.attn_ln.bias", "encoder.layers.3.ffn_layernorm.weight", "encoder.layers.3.ffn_layernorm.bias", "encoder.layers.3.self_attn.c_attn", "decoder.layers.0.self_attn_ln.weight", "decoder.layers.0.self_attn_ln.bias", "decoder.layers.0.cross_attn_ln.weight", "decoder.layers.0.cross_attn_ln.bias", "decoder.layers.0.ffn_layernorm.weight", "decoder.layers.0.ffn_layernorm.bias", "decoder.layers.0.self_attn.c_attn", "decoder.layers.0.encoder_attn.c_attn", "decoder.layers.1.self_attn_ln.weight", "decoder.layers.1.self_attn_ln.bias", "decoder.layers.1.cross_attn_ln.weight", "decoder.layers.1.cross_attn_ln.bias", "decoder.layers.1.ffn_layernorm.weight", "decoder.layers.1.ffn_layernorm.bias", "decoder.layers.1.self_attn.c_attn", "decoder.layers.1.encoder_attn.c_attn", "decoder.layers.2.self_attn_ln.weight", "decoder.layers.2.self_attn_ln.bias", "decoder.layers.2.cross_attn_ln.weight", "decoder.layers.2.cross_attn_ln.bias", "decoder.layers.2.ffn_layernorm.weight", "decoder.layers.2.ffn_layernorm.bias", "decoder.layers.2.self_attn.c_attn", "decoder.layers.2.encoder_attn.c_attn", "decoder.layers.3.self_attn_ln.weight", "decoder.layers.3.self_attn_ln.bias", "decoder.layers.3.cross_attn_ln.weight", "decoder.layers.3.cross_attn_ln.bias", "decoder.layers.3.ffn_layernorm.weight", "decoder.layers.3.ffn_layernorm.bias", "decoder.layers.3.self_attn.c_attn", "decoder.layers.3.encoder_attn.c_attn".
:
During handling of the above exception, another exception occurred:

Error(s) in loading state_dict for OFAModel:
Unexpected key(s) in state_dict: "encoder.layers.0.attn_ln.weight", "encoder.layers.0.attn_ln.bias", "encoder.layers.0.ffn_layernorm.weight", "encoder.layers.0.ffn_layernorm.bias", "encoder.layers.0.self_attn.c_attn", "encoder.layers.1.attn_ln.weight", "encoder.layers.1.attn_ln.bias", "encoder.layers.1.ffn_layernorm.weight", "encoder.layers.1.ffn_layernorm.bias", "encoder.layers.1.self_attn.c_attn", "encoder.layers.2.attn_ln.weight", "encoder.layers.2.attn_ln.bias", "encoder.layers.2.ffn_layernorm.weight", "encoder.layers.2.ffn_layernorm.bias", "encoder.layers.2.self_attn.c_attn", "encoder.layers.3.attn_ln.weight", "encoder.layers.3.attn_ln.bias", "encoder.layers.3.ffn_layernorm.weight", "encoder.layers.3.ffn_layernorm.bias", "encoder.layers.3.self_attn.c_attn", "decoder.layers.0.self_attn_ln.weight", "decoder.layers.0.self_attn_ln.bias", "decoder.layers.0.cross_attn_ln.weight", "decoder.layers.0.cross_attn_ln.bias", "decoder.layers.0.ffn_layernorm.weight", "decoder.layers.0.ffn_layernorm.bias", "decoder.layers.0.self_attn.c_attn", "decoder.layers.0.encoder_attn.c_attn", "decoder.layers.1.self_attn_ln.weight", "decoder.layers.1.self_attn_ln.bias", "decoder.layers.1.cross_attn_ln.weight", "decoder.layers.1.cross_attn_ln.bias", "decoder.layers.1.ffn_layernorm.weight", "decoder.layers.1.ffn_layernorm.bias", "decoder.layers.1.self_attn.c_attn", "decoder.layers.1.encoder_attn.c_attn", "decoder.layers.2.self_attn_ln.weight", "decoder.layers.2.self_attn_ln.bias", "decoder.layers.2.cross_attn_ln.weight", "decoder.layers.2.cross_attn_ln.bias", "decoder.layers.2.ffn_layernorm.weight", "decoder.layers.2.ffn_layernorm.bias", "decoder.layers.2.self_attn.c_attn", "decoder.layers.2.encoder_attn.c_attn", "decoder.layers.3.self_attn_ln.weight", "decoder.layers.3.self_attn_ln.bias", "decoder.layers.3.cross_attn_ln.weight", "decoder.layers.3.cross_attn_ln.bias", "decoder.layers.3.ffn_layernorm.weight", "decoder.layers.3.ffn_layernorm.bias", "decoder.layers.3.self_attn.c_attn", "decoder.layers.3.encoder_attn.c_attn". Traceback (most recent call last):

During handling of the above exception, another exception occurred:

File "../../train.py", line 537, in
Traceback (most recent call last):
File "../../train.py", line 537, in
cli_main()
File "../../train.py", line 530, in cli_main
cli_main()
File "../../train.py", line 530, in cli_main
cli_main()
File "../../train.py", line 530, in cli_main
cli_main()
File "../../train.py", line 530, in cli_main
distributed_utils.call_main(cfg, main)
File "/mypath/fairseq/fairseq/distributed/utils.py", line 374, in call_main
distributed_utils.call_main(cfg, main)
File "/mypath/fairseq/fairseq/distributed/utils.py", line 374, in call_main
distributed_utils.call_main(cfg, main)
File "/mypath/fairseq/fairseq/distributed/utils.py", line 374, in call_main
distributed_utils.call_main(cfg, main)
File "/mypath/fairseq/fairseq/distributed/utils.py", line 374, in call_main
distributed_main(cfg.distributed_training.device_id, main, cfg, kwargs)
File "/mypath/fairseq/fairseq/distributed/utils.py", line 348, in distributed_main
distributed_main(cfg.distributed_training.device_id, main, cfg, kwargs)
distributed_main(cfg.distributed_training.device_id, main, cfg, kwargs)
distributed_main(cfg.distributed_training.device_id, main, cfg, kwargs) File "/mypath/fairseq/fairseq/distributed/utils.py", line 348, in distributed_main
File "/mypath/fairseq/fairseq/distributed/utils.py", line 348, in distributed_main

File "/mypath/fairseq/fairseq/distributed/utils.py", line 348, in distributed_main
main(cfg, **kwargs)
File "../../train.py", line 170, in main
main(cfg, **kwargs)
File "../../train.py", line 170, in main
main(cfg, **kwargs)
File "../../train.py", line 170, in main
disable_iterator_cache=True,
File "/mypath/utils/checkpoint_utils.py", line 254, in load_checkpoint
main(cfg, **kwargs)
File "../../train.py", line 170, in main
disable_iterator_cache=True,disable_iterator_cache=True,

File "/mypath/utils/checkpoint_utils.py", line 254, in load_checkpoint
File "/mypath/utils/checkpoint_utils.py", line 254, in load_checkpoint
reset_meters=reset_meters,
File "/mypath/trainer.py", line 533, in load_checkpoint
disable_iterator_cache=True,
File "/mypath/utils/checkpoint_utils.py", line 254, in load_checkpoint
reset_meters=reset_meters,
reset_meters=reset_meters,
File "/mypath/trainer.py", line 533, in load_checkpoint
File "/mypath/trainer.py", line 533, in load_checkpoint
"please ensure that the architectures match.".format(filename)
Exception: Cannot load model parameters from checkpoint ../../scripts/biomedgpt_tiny.pt; please ensure that the architectures match.
reset_meters=reset_meters,
File "/mypath/trainer.py", line 533, in load_checkpoint
"please ensure that the architectures match.".format(filename)"please ensure that the architectures match.".format(filename)

ExceptionException: : Cannot load model parameters from checkpoint ../../scripts/biomedgpt_tiny.pt; please ensure that the architectures match.Cannot load model parameters from checkpoint ../../scripts/biomedgpt_tiny.pt; please ensure that the architectures match.

"please ensure that the architectures match.".format(filename)

Exception: Cannot load model parameters from checkpoint ../../scripts/biomedgpt_tiny.pt; please ensure that the architectures match.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 239) of binary: /root/miniconda3/envs/biomedgpt/bin/python3
Traceback (most recent call last):
File "/root/miniconda3/envs/biomedgpt/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/root/miniconda3/envs/biomedgpt/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/root/miniconda3/envs/biomedgpt/lib/python3.7/site-packages/torch/distributed/launch.py", line 195, in
main()
File "/root/miniconda3/envs/biomedgpt/lib/python3.7/site-packages/torch/distributed/launch.py", line 191, in main
launch(args)
File "/root/miniconda3/envs/biomedgpt/lib/python3.7/site-packages/torch/distributed/launch.py", line 176, in launch
run(args)
File "/root/miniconda3/envs/biomedgpt/lib/python3.7/site-packages/torch/distributed/run.py", line 756, in run
)(*cmd_args)
File "/root/miniconda3/envs/biomedgpt/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 132, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/root/miniconda3/envs/biomedgpt/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 248, in launch_agent
failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

../../train.py FAILED

Failures:
[1]:
time : 2024-09-23_02:08:13
host : 7ctiahmo5rmk1-0
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 240)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
time : 2024-09-23_02:08:13
host : 7ctiahmo5rmk1-0
rank : 2 (local_rank: 2)
exitcode : 1 (pid: 241)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
time : 2024-09-23_02:08:13
host : 7ctiahmo5rmk1-0
rank : 3 (local_rank: 3)
exitcode : 1 (pid: 242)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

Root Cause (first observed failure):
[0]:
time : 2024-09-23_02:08:13
host : 7ctiahmo5rmk1-0
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 239)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

[taokz]

@aaaaaannie Apologies I missed your response earlier. It seems that arch=ofa_tiny might not be set up correctly. Which checkpoint did you download? We currently have three models available publicly, and if you downloaded the base model, you’ll need to set arch=ofa_base instead.