Torch.cat error

[yimjinkyu1]

I want to insert residual layer to neural net.
But I notice below bug in amp.

Out += torch.cat((shortcut, padding),1)

RuntimeError: Expected object of scalar type Half but got scalar type Float for sequence element 1 in sequence argument at position #1 ‘tensors’

[ptrblck]

Hi @yimjinkyu1,

could you post a small reproducible code snippet so that we can have a look?
This dummy example of a residual connection seems to work:

import torch
import torch.nn as nn

from apex import amp

class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.conv = nn.Conv2d(3, 3, 3, 1, 1)

    def forward(self, x):
        identity = x
        out = self.conv(x)
        out += identity

        return out

device = 'cuda'
model = MyModel().to(device)
x = torch.randn(1, 3, 24, 24, device=device)

model = amp.initialize(model, opt_level='O1')
output = model(x)

[mcarilli]

I think it's the cat call that's failing, not the residual connection. Is that correct?
Cat is on the sequence_casts list so it should take all inputs and cast them to the widest type (in this case fp32).