T5 fp16 forward yields nan

[binshengliu]

🐛 Bug

Information

Model I am using (Bert, XLNet ...): T5

Language I am using the model on (English, Chinese ...): English

The problem arises when using:

• the official example scripts: (give details below)
• my own modified scripts: (give details below)

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)
• my own task or dataset: (give details below)

To reproduce

I use pytorch-lightning to manage fp16. This is the minimal example that reproduces the result.

from transformers import T5Model, T5Tokenizer

tokenizer = T5Tokenizer.from_pretrained("t5-base")
model = T5Model.from_pretrained("t5-base").cuda().half()
text = "hello world!"
inputs = tokenizer.encode(text, return_tensors="pt").cuda()
out = model(input_ids=inputs, decoder_input_ids=inputs)
print(out[0][:, :, :10])

output:

tensor([[[nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
         [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
         [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan]]], device='cuda:0',
       dtype=torch.float16, grad_fn=<SliceBackward>)

Expected behavior

Get non-nan values.

Environment info

• transformers version: 2.9.0
• Platform: Linux-4.15.0-88-generic-x86_64-with-debian-buster-sid
• Python version: 3.7.6
• PyTorch version (GPU?): 1.4.0 (True)
• Tensorflow version (GPU?): not installed (NA)
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No

[patrickvonplaten]

Thanks for the detailed error description @binshengliu! I linked a PR that should fix it :-)

[binshengliu]

Hi, there's still some chance we get nan values during training. It does not always happen but is still pretty easy to reproduce. My current version is 2.11.0.

import torch
from transformers import T5Model, T5Tokenizer

torch.manual_seed(0)
tokenizer = T5Tokenizer.from_pretrained("t5-base")
model = T5Model.from_pretrained("t5-base").cuda().half()
text = "hello world!"
model.train()
inputs = tokenizer.encode(text, return_tensors="pt").cuda()
for idx in range(1000):
    out = model(input_ids=inputs, decoder_input_ids=inputs)
    if torch.isnan(out[0]).any():
        print(idx)
        print(out[0][:, :, :10])
        exit()

Output:

143
tensor([[[nan, nan, nan, nan, nan, nan, nan, nan, nan, 0.],
         [nan, nan, nan, 0., nan, nan, nan, nan, nan, 0.],
         [nan, nan, 0., nan, nan, 0., 0., 0., nan, nan]]], device='cuda:0',
       dtype=torch.float16, grad_fn=<SliceBackward>)

[binshengliu]

Sorry I just noticed you commented #4586 (comment). I can confirm the issue I encountered is caused by the same reason.

[yeliu918]

Hi, I have the same problem. I get NAN when using fp16. And when I set fp16=False, the NAN problem is gone.

[patrickvonplaten]

Yeah that's still an open problem....not sure how easy it will be to solve it, see: #4586 (comment)

[SamsTheGreatest]

Same when fine-tuning GPT Neo.