Fixing torch.ctc err

[teowenshen]

F.ctc_loss throws error when batch size N is 1. Using torch.long as recommended by the documentation fixes this error.

import torch
N = 1
T = 50
S = 31
V = 20
log_probs = torch.randn(T, N, V, device="cuda:0").log_softmax(2)
targets = torch.randint(1, 20, (S,), device="cuda:0", dtype=torch.int32)
input_lengths = torch.full((N,), T, device="cuda:0", dtype=torch.int32)
target_lengths = torch.tensor([S,], device="cuda:0", dtype=torch.int32)

torch.nn.functional.ctc_loss(log_probs, targets, input_lengths, target_lengths)

throws:

RuntimeError: Expected tensor to have CPU Backend, but got tensor with CUDA Backend (while checking arguments for cudnn_ctc_loss)

[csukuangfj]

The error is from the following line
https://github.com/pytorch/pytorch/blob/2b48891e62e5c4b57c8cac92cee5eb71228a203a/aten/src/ATen/native/cudnn/LossCTC.cpp#L146

  checkBackend(c, {*targets}, Backend::CPU);

You can see that it expects targets on CPU.

---

As for the lengths, https://github.com/pytorch/pytorch/blob/2b48891e62e5c4b57c8cac92cee5eb71228a203a/aten/src/ATen/native/cudnn/LossCTC.cpp#L221

  Tensor ilc = input_lengths.to(Device(at::kCPU), at::kLong).contiguous();
  Tensor tlc = target_lengths.to(Device(at::kCPU), at::kLong).contiguous();

They are converted to kLong and moved to CPU.

So the only thing that needs to be changed is to move targets to CPU.

[yfyeung]

Disable cudnn also works for me.

        with torch.backends.cudnn.flags(enabled=False):
            ctc_loss = torch.nn.functional.ctc_loss(
                log_probs=ctc_output.permute(1, 0, 2),  # (T, N, C)
                targets=targets,
                input_lengths=encoder_out_lens,
                target_lengths=target_lengths,
                reduction="sum",
            )

[teowenshen]

I have followed @csukuangfj 's advice and moved the targets and lengths to CPU. There were some comments that say cudnn is the only deterministic implementation in PyTorch.

Surprisingly, in my case the error wasn't thrown until when I had a last batch in the validation loop that had only 1 sample. But it doesn't seem to be related to the batch size after all.

[csukuangfj]

Thanks!