test_handler_metrics_saver_dist error

[wyli]

Describe the bug
a frequent error from the premerge tests, e.g.

• https://github.com/Project-MONAI/MONAI/runs/4753115139?check_suite_focus=true
• https://github.com/Project-MONAI/MONAI/runs/4773096722?check_suite_focus=true

tests/test_handler_metrics_saver_dist.py
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/__w/MONAI/MONAI/tests/test_handler_metrics_saver_dist.py", line 87, in _run
    for i, row in enumerate(f_csv):
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/__w/MONAI/MONAI/tests/utils.py", line 396, in run_process
    raise e
  File "/__w/MONAI/MONAI/tests/utils.py", line 387, in run_process
    func(*args, **kwargs)
  File "/__w/MONAI/MONAI/tests/utils.py", line 557, in _call_original_func
    return f(*args, **kwargs)
  File "/__w/MONAI/MONAI/tests/test_handler_metrics_saver_dist.py", line 31, in test_content
    self._run(tempdir)
  File "/__w/MONAI/MONAI/tests/test_handler_metrics_saver_dist.py", line 87, in _run
    for i, row in enumerate(f_csv):
OSError: [Errno 9] Bad file descriptor
F
======================================================================
FAIL: test_content (__main__.DistributedMetricsSaver)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/MONAI/MONAI/tests/utils.py", line 432, in _wrapper
    assert results.get(), "Distributed call failed."
AssertionError: Distributed call failed.

----------------------------------------------------------------------
Ran 1 test in 10.636s

[wyli]

the root cause is that the heavy lifting happens on rank 0, and rank 1 may exit early and clear the process rank information. get_rank() might return a wrong number after exiting and clearing:

MONAI/tests/test_handler_metrics_saver_dist.py

Line 82 in 2fef7ff

if dist.get_rank() == 0:

[wyli]

still an issue https://github.com/Project-MONAI/MONAI/runs/4852155300?check_suite_focus=true

I'll add a barrier following the comment #3641 (comment) cc @Nic-Ma