How distributed training affects performance？

[tanbuzheng]

Hi， your work is great！May I ask you a queastion?
I found that using different ddp settings has a greater impact on the final result. How to overcome this? In other words, how to set the batch size and number of GPUs appropriately?

[fenglinglwb]

We conducted all experiments using 8 GPUs. In my experience, larger batches may help.

[tanbuzheng]

Thanks for your reply! I have another question.
In training_loop.py, I found you used a class "InfiniteSampler" as training sampler. Is there any difference between this and torch.utils.data.distributed.DistributedSampler?

[fenglinglwb]

Actually, I didn't dive into this point. We mainly build our framework based on the StyleGAN-ada code.

[tanbuzheng]

OK, I see. Thanks a lot.