Iterations completing out of order (possibly) in ddp with torchelastic?

[jloveric]

This might be bug or might be expected. I'm running a pytorchlightning with torchelastic and ddp. I'm noticing the iterations are being dumped out of order (below iteration 632 preceeds iteration 574). This could be due to delays in parallel writing... or perhaps just issues in logging. Is this expected behavior?

Validating: 60it [00:21,  3.61it/s]�[A
Epoch 26: : 632it [08:13,  1.28it/s, loss=0.111, v_num=0]

Validating: 62it [00:22,  4.62it/s]�[A
Validating: 0it [00:00, ?it/s]�[A
Epoch 26: : 572it [07:51,  1.21it/s, loss=0.111, v_num=0]

Validating: 2it [00:00, 18.62it/s]�[A
Epoch 26: : 574it [07:52,  1.22it/s, loss=0.111, v_num=0]

Running with 6 gpus in ddp.

[awaelchli]

This happens with torchelastic only?
can we check that the rank is correctly set, by printing trainer.global_rank somewhere in the training_step for example? I suspect these progress bars are from different ranks. It should only show on rank 0.
which PL version?

[edenlightning]

@jloveric mind giving more details?