DataLoader tutorial does not handle num_workers > 0

[NicolasHug]

I just wanted to document an issue with the tutorials https://pytorch.org/data/beta/tutorial.html#working-with-dataloader

The code in the tutorial will not work when running multiple DataLoader processes as the datapipe will be duplicated across workers:

    dl = DataLoader(dataset=datapipe, batch_size=2, shuffle=True, num_workers=2)

    for i, e in enumerate(dl):
        print(e)

gives

{'label': tensor([7, 0], dtype=torch.int32), 'data': tensor([[0.5105, 0.7899],
        [0.0152, 0.5981]], dtype=torch.float64)}
{'label': tensor([7, 0], dtype=torch.int32), 'data': tensor([[0.5105, 0.7899],
        [0.0152, 0.5981]], dtype=torch.float64)}
{'label': tensor([4, 6], dtype=torch.int32), 'data': tensor([[0.9998, 0.5452],
        [0.8515, 0.8264]], dtype=torch.float64)}
{'label': tensor([4, 6], dtype=torch.int32), 'data': tensor([[0.9998, 0.5452],
        [0.8515, 0.8264]], dtype=torch.float64)}
{'label': tensor([1, 9], dtype=torch.int32), 'data': tensor([[0.8423, 0.3664],
        [0.6397, 0.6408]], dtype=torch.float64)}
{'label': tensor([1, 9], dtype=torch.int32), 'data': tensor([[0.8423, 0.3664],
        [0.6397, 0.6408]], dtype=torch.float64)}
...

Even though this is still beta, it may still be worth letting users know about such pitfalls.

Also, since there are various ways to achieve the sharding, it could be useful to settle on a definite canonical way of handling all this.

[ejguan]

Thanks for the feedback!!!

Also, since there are various ways to achieve the sharding, it could be useful to settle on a definite canonical way of handling all this.

Yeah. Like where users should put sharding in the datapipe graph. Also, the discrepancy of result should be mentioned as well in #302

[franchesoni]

It would be nice to see an updated tutorial, I don't know how to do this in an easy way 😭

Using 0 < num_workers is a very common usecase, it could be nice to have a default solution. On top, the datapipe examples provided have no consideration for this case (except https://github.com/pytorch/torchrec/blob/main/torchrec/datasets/criteo.py).

Maybe a quick explanation here could suffice for now 😜

[NivekT]

The tutorial has been updated. If anyone is still experiencing any issue with the nightly version, feel free to re-open.