The error in new machine when running distribution/fedavg

[weiyikang]

I think the FedML can use easily on another machine only by cloning the FedML without modifications. While the errors occur as follows:

Configures of computer:

1. 4 * RTX 3090, cuda 11.1
2. pytorch1.7
3. According to CI-install.sh requiring the environment

run the command:
sh run_fedavg_distributed_pytorch.sh 4 4 1 4 cnn homo 2 1 32 0.0001 digit5 "./../../../data/Digit5" 0
There are 4 clients and 4 works.

The errors as follows(Fig.1 the warning of program, Fig.2 the 4 clients on GPUs may be wrong, the same process on all GPUs):

[chaoyanghe]

are you using the same version of the code? We updated a lot recently.

[weiyikang]

I used the previous version, I will try the latest version again.

[weiyikang]

I have used the latest version, FedAvg(distribution) on 4 * RTX3090 computer as shown in Fig.1, while the previous version on 8 * RTX2080 as shown in Fig.2.

1. Why there exist many 0MiB FedAvg(distributed) in the latest version?
2. What's the effect of 0MiB FedAvg(distributed)?
3. Why the same process(eg. Process FedAvg(distributed):1 ) in different GPUs, I think the same process should only load on one GPU.
4. Whether this phenomenon in Fig.1 caused by the software environment, not the latest version's features?

[chaoyanghe]

May I know how you set your "init_training_device" function?