Add distributed training sage e2e example on homogeneous graphs [5/6]

[ZhengHongming888]

This code belongs to the part of the whole distributed training for PyG.

You can run this example as command below in 2 nodes (2 partitions) -

This homo e2e example will base on ogbn-product dataset with 2 partitions which is generate by our partition example script in the same folder. The default number of sampler workers is 2 and concurrency for mp.queue is 2.. And you can also setup other number to give you more gain on throughput.

node 0:
python dist_train_sage_for_homo.py --dataset_root_dir=your partition folder --num_nodes=2 --node_rank=0 --num_training_procs=1 --master_addr= master ip

node 1:
python dist_train_sage_for_homo.py --dataset_root_dir=your partition folder --num_nodes=2 --node_rank=1 --num_training_procs=1 --master_addr= master ip

The training will output the progress bar as the status like
Epoch 00: 27%|████████████████████████████████▊ | 26624/98307 [ ]

After training/test epochs there will generate 3 result files -

1. dist_train_sage_for_homo.txt - general training information /arguments
2. dist_train_sage_for_homo_rank0.txt - results for training loss, accuracy, epoch time on rank0 node
3. dist_train_sage_for_homo_rank1.txt - results for training loss, accuracy, epoch time on rank1 node

You also can reference from our e2e README for more distributed example cases and will add more like hetero, edge_sampler, etc and their running command. Welcome for any comments!

Thanks.

[JakubPietrakIntel]

@ZhengHongming888 I left a request to remove barrier waits and enable persistent workers instead.

[JakubPietrakIntel]

#8713 adds an updated version of e2e examples for dist PyG, so I'm closing this PR in agreement with @ZhengHongming888