Added instructions and script for distributed training with Horovod

[tlkh]

Added a script to show how to use Horovod to do multi-GPU or distributed training across machines.

Also added instructions and sample command in README.md

In addition, fixed a small type on README training command (./train should be ./train.py)

[nshepperd]

Awesome, thanks!

[XinyuDu]

Added a script to show how to use Horovod to do multi-GPU or distributed training across machines.

Also added instructions and sample command in README.md

In addition, fixed a small type on README training command (./train should be ./train.py)

Please consider using memory_saving_gradients for retraining 345M with horovod. THX.

[shamiul94]

I tried to run train_horovod.py on a custom dataset using this command.

mpirun -np 4 -H localhost:4 -bind-to none -map-by slot -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH -x PYTHONPATH=src -mca pml ob1 -mca btl ^openib python /home/ec2-user/SageMaker/gpt-2/train-horovod.py --dataset /home/ec2-user/SageMaker/gpt-2/src/Dataset/data.npz --model_name /home/ec2-user/SageMaker/gpt-2/src/models/345M --batch_size 1

This got OOM error. It could not allocate tensors of shape [12240, 840] and similar. I was using four 16GB V100 GPUs on AWS Sagemaker (aws ml.p3.8xlarge). What is going wrong here? Am I not using enough GPUs or is it not possible to multi-GPU train 345M?
I explained this problem in issue #53.