Multi-GPU autograd error with Pytorch 0.4

[erogol]

After updating pytorch 0.4 I am getting the following error when I try to train my model here: https://github.com/mozilla/TTS with multi-gpus. I have no idea about what it means unfortunately. A bug or just a problem that I need some feedback on. Thx.

Traceback (most recent call last):
  File "train.py", line 403, in <module>
    main(args)
  File "train.py", line 393, in main
    model, criterion, train_loader, optimizer, epoch)
  File "train.py", line 111, in train
    model.forward(text_input, mel_spec)
  File "/home/erogol/miniconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 114, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/erogol/miniconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 124, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/erogol/miniconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 65, in parallel_apply
    raise output
  File "/home/erogol/miniconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 41, in _worker
    output = module(*input, **kwargs)
  File "/home/erogol/miniconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/erogol/projects/TTS/models/tacotron.py", line 28, in forward
    encoder_outputs = self.encoder(inputs)
  File "/home/erogol/miniconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/erogol/projects/TTS/layers/tacotron.py", line 205, in forward
    return self.cbhg(inputs)
  File "/home/erogol/miniconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/erogol/projects/TTS/layers/tacotron.py", line 183, in forward
    outputs, _ = self.gru(x)
  File "/home/erogol/miniconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/erogol/miniconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 192, in forward
    output, hidden = func(input, self.all_weights, hx, batch_sizes)
  File "/home/erogol/miniconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/_functions/rnn.py", line 323, in forward
    return func(input, *fargs, **fkwargs)
  File "/home/erogol/miniconda3/envs/pytorch4/lib/python3.6/site-packages/torch/nn/_functions/rnn.py", line 287, in forward
    dropout_ts)
RuntimeError: torch/csrc/autograd/variable.cpp:115: get_grad_fn: Assertion `output_nr == 0` failed.

[ssnl]

Urr. This seems like a bug. Can you try to come up with a minimal working example please? Thanks for reporting!

[erogol]

The problem is flattening the parameters. If I dont use it everything works with a warning suggesting it.

import torch

class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.gru = torch.nn.GRU(128, 64, 1, batch_first=True, bidirectional=True)
                                
    def forward(self, inp):
        self.gru.flatten_parameters()
        out, _ = self.gru(inp)
        return out
                        

net = Net()
inp = torch.rand(32, 8, 128, requires_grad=True)

net = torch.nn.DataParallel(net)

inp = inp.cuda()
net = net.cuda()
out = net.forward(inp)

[ssnl]

@erogol Thanks Eren. This is very helpful. We'll look into it.

[zou3519]

@erogol What was the last version of pytorch you used that the code worked? The code crashes on 0.3.1 for me

[erogol]

@zou3519 0.3.0

[adampolyak]

Same bug when using the code here: https://github.com/NVIDIA/tacotron2 with Pytorch0.4

[BangLiu]

I also have the same problem here. Has the bug been fixed? Or do we have any solution now?

[colesbury]

@BangLiu the workaround is to to remove the flatten_parameters() call from def forward(). I'm not sure how that ever worked in a DataParallel.

[erogol]

@colesbury do you think that would make any performance difference, calling it in forward()

[soumith]

@erogol yes, calling it in forward is detrimental to performance.

If you are using RNN, it is better to use DistributedDataParallel (and 1 process per GPU) than using DataParallel. It has benefits of being faster and you dont have to restructure your batching (your code is as if it is just using 1-GPU).

See https://pytorch.org/docs/stable/distributed.html#launch-utility

[liqing-ustc]

+1. I met the same bug.

[soumith]

@liqing-ustc see #7092 (comment) for answer

[wangkenpu]

I met the same problem with PyTorch 0.4.0. I was wondering if this bug have been solved?

I found if I deleted self.gru.flatten_parameters() in forward() function, every thing worked well. But I faced another warning.

UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters().