Inf Loss Problem When Training

[nreHieW]

In line 1628-1629 of transformers/src/transformers/models/chameleon/modeling_chameleon.py

image_tokens = self.model.vocabulary_mapping.image_tokens
logits[:, :, image_tokens] = torch.finfo(logits.dtype).min

My understanding is that this is here because the original Chameleon intentionally did not want to generate any image tokens. But keeping this in for training would lead to inf loss.

Is there an updated version of the code?

[leloykun]

Hi! You can either use the version in the transformers folder or this PR of mine to the main Transformers library: huggingface/transformers#32013