Backward pass error during OWL-ViT finetuning

[pasqualedem]

System Info

Google Colab

Who can help?

@ArthurZucker
@amyeroberts

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

https://colab.research.google.com/drive/1GfspAHLGTMmzfNShAVebzM04s32q1LKb?usp=sharing

Expected behavior

I was trying to finetune Owl-ViT, and I came across this backward pass error. I investigated and found out that the error is the operator "/=" which does an inplace operation.
At lines:

• transformers/src/transformers/models/owlvit/modeling_owlvit.py

  Line 1296 in 4e8929d

  image_class_embeds /= torch.linalg.norm(image_class_embeds, dim=-1, keepdim=True) + 1e-6

• transformers/src/transformers/models/owlvit/modeling_owlvit.py

  Line 1297 in 4e8929d

  query_embeds /= torch.linalg.norm(query_embeds, dim=-1, keepdim=True) + 1e-6

[ydshieh]

@pasqualedem

Would you like to open a PR to fix it ? 🤗

[Pro-flynn]

Could you share your scripts for owlvit finetuning? Thanks @pasqualedem