Error(s) in loading state_dict for ControlLDM

[yumemiso]

Thank you for your impressive work.
When I use "FreeDoM-CN-style/faceID" example and run python pose2image.py --seed 1234 --timesteps 100 --prompt "young man, realitic photo" --pose_ref "./test_imgs/pose4.jpg" --id_ref "./test_imgs/id3.png" someting errors happened:

Loaded model config from [./models/cldm_v15.yaml]
Loaded state_dict from [./models/control_sd15_openpose.pth]
Traceback (most recent call last):
  File "/home/user2/models/FreeDoM-main/CN/pose2image.py", line 31, in <module>
    model.load_state_dict(load_state_dict('./models/control_sd15_openpose.pth', location='cuda'))
  File "/home/user2/anaconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2152, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for ControlLDM:
        Unexpected key(s) in state_dict: "cond_stage_model.transformer.text_model.embeddings.position_ids". 

It seems faild to load SD pre-trained model parameters, then I try to put model in cuda before load parameters.

model = create_model('./models/cldm_v15.yaml').cpu()
model = model.cuda()
model.load_state_dict(load_state_dict('./models/control_sd15_openpose.pth', location='cuda'))

but the another error occurred:

Loaded model config from [./models/cldm_v15.yaml]
Loaded state_dict from [./models/control_sd15_openpose.pth]
Traceback (most recent call last):
  File "/home/user2/models/FreeDoM-main/CN/pose2image.py", line 35, in <module>
    ddim_sampler = DDIMSampler(model, add_condition_mode="face_id", ref_path=args.pose_ref, add_ref_path=args.id_ref, no_freedom=args.no_freedom)
  File "/home/user2/models/FreeDoM-main/CN/cldm/ddim_hacked.py", line 128, in __init__
    self.idloss = IDLoss(ref_path=add_ref_path).cuda()
  File "/home/user2/models/FreeDoM-main/CN/cldm/arcface/model.py", line 12, in __init__
    self.facenet.load_state_dict(torch.load("cldm/arcface/model_ir_se50.pth"))
  File "/home/user2/anaconda3/lib/python3.10/site-packages/torch/serialization.py", line 1028, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/home/user2/anaconda3/lib/python3.10/site-packages/torch/serialization.py", line 1231, in _legacy_load
    return legacy_load(f)
  File "/home/user2/anaconda3/lib/python3.10/site-packages/torch/serialization.py", line 1117, in legacy_load
    tar.extract('storages', path=tmpdir)
  File "/home/user2/anaconda3/lib/python3.10/tarfile.py", line 2081, in extract
    tarinfo = self.getmember(member)
  File "/home/user2/anaconda3/lib/python3.10/tarfile.py", line 1803, in getmember
    raise KeyError("filename %r not found" % name)
KeyError: "filename 'storages' not found"

I don't know how to solve this. Is it because I'm using the wrong pre-trained model? It is downloaded from lllyasviel/ControlNet at main (huggingface.co).

[ymuang]

你好，我遇到了相同的错误，请问您这个问题怎么解决的 Hello, I have encountered the same error. How did you resolve this issue?

[ymuang]

你好，我遇到了相同的错误，请问您这个问题怎么解决的 Hello, I have encountered the same error. How did you resolve this issue?

model.load_state_dict(load_state_dict('./models/control_sd15_openpose.pth', location='cuda'),strict=False) 这样貌似解决了