Skip to content

Commit d1f33ac

Browse files
fco-dvsdesrozis
fco-dv
and
sdesrozis
authored
Docker for users with MSDeepSpeed (#1304)
* Docker for users with DeepSpeed - msdp-base | vision | nlp * Docker for users with DeepSpeed - rename images extensions to msdp-apex-* Co-authored-by: Sylvain Desroziers <sylvain.desroziers@gmail.com>
1 parent 4578af1 commit d1f33ac

File tree

6 files changed

+114
-7
lines changed

6 files changed

+114
-7
lines changed

‎README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -308,11 +308,11 @@ docker run --gpus all -it -v $PWD:/workspace/project --network=host --shm-size 1
308308

309309
Available pre-built images are :
310310

311-
- `pytorchignite/base:latest | pytorchignite/hvd-base:latest`
311+
- `pytorchignite/base:latest | pytorchignite/hvd-base:latest | pytorchignite/msdp-apex-base:latest`
312312
- `pytorchignite/apex:latest | pytorchignite/hvd-apex:latest`
313-
- `pytorchignite/vision:latest | pytorchignite/hvd-vision:latest`
313+
- `pytorchignite/vision:latest | pytorchignite/hvd-vision:latest | pytorchignite/msdp-apex-vision:latest`
314314
- `pytorchignite/apex-vision:latest | pytorchignite/hvd-apex-vision:latest`
315-
- `pytorchignite/nlp:latest | pytorchignite/hvd-nlp:latest`
315+
- `pytorchignite/nlp:latest | pytorchignite/hvd-nlp:latest | pytorchignite/msdp-apex-nlp:latest`
316316
- `pytorchignite/apex-nlp:latest | pytorchignite/hvd-apex-nlp:latest`
317317

318318
For more details, see [here](docker).

‎docker/README.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,13 @@ Available Tensor Operations:
5050
* `docker pull pytorchignite/hvd-apex-vision:latest`
5151
- [hvd/Dockerfile.hvd-apex-nlp](hvd/Dockerfile.hvd-apex-nlp): base Horovod apex with useful NLP libraries
5252
* `docker pull pytorchignite/hvd-apex-nlp:latest`
53-
53+
- [msdp/Dockerfile.msdp-apex-base](msdp/Dockerfile.msdp-apex-base): multi-stage MSDeepSpeed build with latest Pytorch, Ignite image with minimal dependencies
54+
* `docker pull pytorchignite/msdp-base:latest`
55+
- [msdp/Dockerfile.msdp-apex-vision](msdp/Dockerfile.msdp-apex-vision): base MSDeepSpeed build with useful computer vision libraries
56+
* `docker pull pytorchignite/msdp-vision:latest`
57+
- [msdp/Dockerfile.msdp-apex-nlp](msdp/Dockerfile.msdp-apex-nlp): base MSDeepSpeed build with useful NLP libraries
58+
* `docker pull pytorchignite/msdp-nlp:latest`
59+
5460
## How to use
5561

5662
```bash

‎docker/msdp/Dockerfile.msdp-apex-base‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# Multi-stage build
2+
# 1/Building apex with pytorch:1.6.0-cuda10.1-cudnn7-devel
3+
FROM pytorch/pytorch:1.6.0-cuda10.1-cudnn7-devel AS msdp-builder
4+
5+
ARG ARG_TORCH_CUDA_ARCH_LIST="6.0;6.1;6.2;7.0;7.5"
6+
ENV TORCH_CUDA_ARCH_LIST=$ARG_TORCH_CUDA_ARCH_LIST
7+
8+
# Install git
9+
RUN apt-get update && apt-get install -y --no-install-recommends git && \
10+
rm -rf /var/lib/apt/lists/*
11+
12+
# For pip --use-feature option
13+
RUN python -m pip install --upgrade pip
14+
15+
# From https://github.com/microsoft/DeepSpeed/blob/master/docker/Dockerfile
16+
##############################################################################
17+
# DeepSpeed
18+
##############################################################################
19+
RUN git clone https://github.com/microsoft/DeepSpeed.git /tmp/DeepSpeed
20+
RUN cd /tmp/DeepSpeed && \
21+
git checkout . && \
22+
git checkout master && \
23+
./install.sh --allow_sudo
24+
25+
# Build runtime image
26+
FROM pytorch/pytorch:1.6.0-cuda10.1-cudnn7-runtime
27+
28+
# For building cpufeature wheel
29+
RUN apt-get update && apt-get install -y --no-install-recommends g++ gcc
30+
31+
# Apex
32+
COPY --from=msdp-builder /tmp/DeepSpeed/third_party/apex/dist/apex-*.whl /apex/
33+
RUN cd /apex && \
34+
pip install --no-cache-dir apex-*.whl && \
35+
rm -fr /apex
36+
37+
# MSDeepSpeed
38+
COPY --from=msdp-builder /tmp/DeepSpeed/dist/deepspeed-*.whl /msdp/
39+
RUN cd /msdp && \
40+
pip install --no-cache-dir deepspeed-*.whl && \
41+
rm -fr /msdp
42+
43+
# Install tzdata / git
44+
RUN apt-get update && \
45+
ln -fs /usr/share/zoneinfo/America/New_York /etc/localtime && \
46+
apt-get install -y tzdata && \
47+
dpkg-reconfigure --frontend noninteractive tzdata && \
48+
apt-get -y install --no-install-recommends git && \
49+
rm -rf /var/lib/apt/lists/*
50+
51+
# Ignite main dependencies
52+
RUN pip install --upgrade --no-cache-dir pytorch-ignite \
53+
tensorboard \
54+
tqdm
55+
# Checkout Ignite examples only
56+
RUN mkdir -p pytorch-ignite-examples && \
57+
cd pytorch-ignite-examples && \
58+
git init && \
59+
git config core.sparsecheckout true && \
60+
echo examples >> .git/info/sparse-checkout && \
61+
git remote add -f origin https://github.com/pytorch/ignite.git && \
62+
git pull origin master
63+
64+
WORKDIR /workspace
65+
66+
ENTRYPOINT ["/bin/bash"]
67+
68+
69+
70+

‎docker/msdp/Dockerfile.msdp-apex-nlp‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# Dockerfile.msdp-apex-nlp
2+
FROM pytorchignite/msdp-apex-base:latest
3+
4+
# Ignite NLP dependencies
5+
RUN pip install --upgrade --no-cache-dir torchtext \
6+
transformers \
7+
spacy \
8+
nltk
9+
10+
ENTRYPOINT ["/bin/bash"]

‎docker/msdp/Dockerfile.msdp-apex-vision‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Dockerfile.msdp-apex-vision
2+
FROM pytorchignite/msdp-apex-base:latest
3+
4+
# Install opencv dependencies
5+
RUN apt-get update && \
6+
apt-get -y install --no-install-recommends libglib2.0 \
7+
libsm6 \
8+
libxext6 \
9+
libxrender-dev && \
10+
rm -rf /var/lib/apt/lists/*
11+
12+
# Ignite vision dependencies
13+
RUN pip install --upgrade --no-cache-dir albumentations \
14+
image-dataset-viz \
15+
numpy \
16+
opencv-python \
17+
py_config_runner \
18+
pillow \
19+
"trains>=0.15.0"
20+
21+
ENTRYPOINT ["/bin/bash"]

‎docs/source/index.rst‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -78,11 +78,11 @@ Pull a pre-built docker image from `our Docker Hub <https://hub.docker.com/u/pyt
7878
7979
Available pre-built images are :
8080

81-
- ``pytorchignite/base:latest | pytorchignite/hvd-base:latest``
81+
- ``pytorchignite/base:latest | pytorchignite/hvd-base:latest | pytorchignite/msdp-apex-base:latest``
8282
- ``pytorchignite/apex:latest | pytorchignite/hvd-apex:latest``
83-
- ``pytorchignite/vision:latest | pytorchignite/hvd-vision:latest``
83+
- ``pytorchignite/vision:latest | pytorchignite/hvd-vision:latest | pytorchignite/msdp-apex-vision:latest``
8484
- ``pytorchignite/apex-vision:latest | pytorchignite/hvd-apex-vision:latest``
85-
- ``pytorchignite/nlp:latest | pytorchignite/hvd-nlp:latest``
85+
- ``pytorchignite/nlp:latest | pytorchignite/hvd-nlp:latest | pytorchignite/msdp-apex-nlp:latest``
8686
- ``pytorchignite/apex-nlp:latest | pytorchignite/hvd-apex-nlp:latest``
8787

8888
For more details, see `here <https://github.com/pytorch/ignite/tree/master/docker>`_.

0 commit comments

Comments
 (0)