Enhances process_group_test

[allenwang28]

See #109 for more context, but there was a desire to:

1. Have collectives be tested on their own
2. Test true resiliency behaviors

What does this PR do?

• Removes _test_multi_pg and replaces it with individual run_{collective}_tests
• Makes each run_{collective}_test robust to different world sizes (i.e. 1, 2, or 3) to be more generalizable
• Introduces MultiPgBaseTest, a base test that spins up process groups and threadpool executors on setup (and destroys on teardown) so they can be re-used inbetween each test invocation.
• MultiPgBaseTest introduces:
  • _run_parallel, a convenience function to run a collective test across its process groups, and
  • _run_with_resiliency, a convenience function that runs a collective test, invokes failure modes and tests for correct resiliency behaviors.
• Removes all tests that utilized _test_multi_pg into GlooMultiPgTest, BabyGlooMultiPgTest, BabyNcclMultiPgTest and BabyNcclResiliencyTest
  • Note that BabyNcclMultiPgTest and BabyNcclResiliencyTest are separated so that the former can test for 2 GPUs, and the latter for 3. While all collectives could theoretically be tested with only 2 GPUs, it seems a bit trivial to run with only one process group.

Note that the test time increases from ~17s to ~43.48s:

==============================  60 passed in 43.48s ============================== 

mostly due to the resiliency tests taking awhile to run. Without the resiliency tests it takes 24.3 seconds:

$ pytest torchft/process_group_test.py -k "not resiliency"  && pkill -U $(whoami) py
thon
====================================== test session starts ======================================
platform linux -- Python 3.10.16, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/allencwang/workspace/torchft
configfile: pytest.ini
plugins: typeguard-2.13.3
collected 60 items / 23 deselected / 37 selected                                                

torchft/process_group_test.py .....................................                       [100%]

============================== 37 passed, 23 deselected in 24.29s ==============================