Added instructions for training using Horovod

README.md

@@ -107,6 +107,18 @@ PYTHONPATH=src ./encode.py <file|directory|glob> /path/to/encoded.npz
 PYTHONPATH=src ./train.py --dataset /path/to/encoded.npz
 ```
 
+To do distributed on multiple GPUs or machines using Horovod: 
+
+```
+mpirun -np 4 \
+ -H localhost:4 \
+ -bind-to none -map-by slot \
+ -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH \
+ -x PYTHONPATH=src \
+ -mca pml ob1 -mca btl ^openib \
+ /home/jovyan/gpt-2/train-horovod.py --dataset encoded.npz
+```
+
 ## GPT-2 samples
 
 | WARNING: Samples are unfiltered and may contain offensive content. |