diff --git a/README.md b/README.md
index 758c880d5..9d68c4a75 100644
--- a/README.md
+++ b/README.md
@@ -107,6 +107,18 @@ PYTHONPATH=src ./encode.py <file|directory|glob> /path/to/encoded.npz
 PYTHONPATH=src ./train.py --dataset /path/to/encoded.npz
 ```
 
+To do distributed on multiple GPUs or machines using Horovod: 
+
+```
+mpirun -np 4 \
+    -H localhost:4 \
+    -bind-to none -map-by slot \
+    -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH \
+    -x PYTHONPATH=src \
+    -mca pml ob1 -mca btl ^openib \
+    /home/jovyan/gpt-2/train-horovod.py --dataset encoded.npz
+```
+
 ## GPT-2 samples
 
 | WARNING: Samples are unfiltered and may contain offensive content. |