3686 Skip workflow run if data is empty or the specified epoch_length is 0

[Nic-Ma]

Fixes #3686 .

Description

This PR enhanced the workflow to skip run if data is empty or the specified epoch_length is 0.

Status

Ready

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• Breaking change (fix or new feature that would cause existing functionality to change).
• New tests added to cover the changes.
• Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
• Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
• In-line docstrings updated.
• Documentation updated, tested make html command in the docs/ folder.

[Nic-Ma]

/black

[Nic-Ma]

/build

[SachidanandAlle]

LGTM.. should be good if we have the test case for multi-gpu as well

[Nic-Ma]

Hi @ericspod @SachidanandAlle ,

Thanks for the review.
I spent much time today to add support for multi-gpu training case, I tried many methods, but no ideal method can guarantee our distributed communication logic works fine with less ranks that the setting, it's PyTorch or NCCL logic, we can't dynamically add or reduce ranks. For example, some handlers or metrics need all ranks to run the same logic to all-gather the result, otherwise, it will hang.
I already added unit tests for multi-gpu training and added more message in the warning for the hanging case.

Thanks.

[Nic-Ma]

/black

[Nic-Ma]

/build