RuntimeError Tensor size mismatch after following the instructions mentioned on this repo README

[sajidahmed12]

python test.py --epoch 499 --folder demo/custom-chkpts --pose_folder demo/sample/poses --key_frame_path demo/sample/image/0024.jpg --s1 8 --s2 3 --n_steps 100 --output_dir demo/sample/results --custom_vae demo/custom-chkpts/vae_1499.pth

Traceback (most recent call last):
  File "/home/UserName/user/DreamPose/test.py", line 48, in <module>
    unet = get_unet('CompVis/stable-diffusion-v1-4', "ebb811dd71cdc38a204ecbdd6ac5d580f529fd8c", resolution=512)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/UserName/user/DreamPose/models/unet_dual_encoder.py", line 30, in get_unet
    unet.conv_in.weight[:, 4:] = torch.zeros(unet.conv_in.weight[:, 3:].shape) # new weights initialized to zero
    ~~~~~~~~~~~~~~~~~~~^^^^^^^
RuntimeError: The expanded size of the tensor (10) must match the existing size (11) at non-singleton dimension 1.  Target sizes: [320, 10, 3, 3].  Tensor sizes: [320, 11, 3, 3]

(yolo) UserName@user-Desktop:~/user/DreamPose$ python test.py --epoch 499 --folder demo/custom-chkpts --pose_folder demo/sample/poses  --key_frame_path demo/sample/key_frame.png --s1 8 --s2 3 --n_steps 100 --output_dir demo/sample/results --custom_vae demo/custom-chkpts/vae_1499.pth
Traceback (most recent call last):
  File "/home/altersense_dragon/sajid/DreamPose/test.py", line 48, in <module>
    unet = get_unet('CompVis/stable-diffusion-v1-4', "ebb811dd71cdc38a204ecbdd6ac5d580f529fd8c", resolution=512)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/UserName/user/DreamPose/models/unet_dual_encoder.py", line 30, in get_unet
    unet.conv_in.weight[:, 4:] = torch.zeros(unet.conv_in.weight[:, 3:].shape) # new weights initialized to zero
    ~~~~~~~~~~~~~~~~~~~^^^^^^^
RuntimeError: The expanded size of the tensor (10) must match the existing size (11) at non-singleton dimension 1.  Target sizes: [320, 10, 3, 3].  Tensor sizes: [320, 11, 3, 3]

[jqliu09]

Same problem encountered. It seems to be caused by the commit "Fix weight initialization in models/unet_dual_encoder.py."

[NolenBrolen]

Same problem here, tried checking out older branches but also ran into errors there due to mismatch n the keys in the saved state dictionary and the keys that the model expects.

According to ChatGPT this often happens when the architecture of the model has changed between the time it was saved and the time it's being loaded, which means that either the saved model is outdated, or the code defining the model has been updated since.

[liuzhuangyuan]

+1

[SaharaSheik]

I solved this by making the following changes to in this file models/unet_dual_encoder.py line 30
https://github.com/johannakarras/DreamPose/blob/5bf30b7df70cf6f2e0bb25556c6ff2cbf0f2b1bf/models/unet_dual_encoder.py#L30C1-L30C117

        unet.conv_in.weight[:, 4:] = torch.zeros(unet.conv_in.weight[:, 3:].shape) # new weights initialized to zero

change to

        unet.conv_in.weight[:, 4:] = torch.zeros(unet.conv_in.weight[:, 4:].shape) # new weights initialized to zero