generate synthetic data for multi-node DDP

[hw-ju]

Hi there,
In the tutorial script unet_training_ddp.py, it seems all local_rank = 0 at each node will generate synthetic data to the same directory when we do multi-node training. Is it necessary and safe? Or shall we only let global_rank = 0 to generate data?

[KumoLiu]

Hi @hw-ju, when doing multi-node training, I think the synthetic data will be generated to the same directory but on different nodes, so it's necessary.
Hope it can help you, thanks!