Skip to content

Commit

Permalink
Update docker instructions (#1127)
Browse files Browse the repository at this point in the history
* update docker instructions

* update default python to 3.7

* update readme

* enable windows use

* enable docs on docker

* update Docker and add jupyter + powershell script

* fix docs building docker info

* fixes

Co-authored-by: ahartikainen <ahartikainen@github.com>
  • Loading branch information
ahartikainen and ahartikainen authored Apr 9, 2020
1 parent c2b80b7 commit 3352a8a
Show file tree
Hide file tree
Showing 10 changed files with 181 additions and 52 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
* Fixed hist kind of `plot_dist` with multidimensional input (#1115)
* Fixed `TypeError` in `transform` argument of `plot_density` and `plot_forest` when `InferenceData` is a list or tuple (#1121)
* Fixed overlaid pairplots issue (#1135)
* Update Docker building steps (#1127)
### Deprecation

### Documentation
Expand Down
49 changes: 43 additions & 6 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ to ensure that any new contribution does not strongly overlap with existing func
## Steps before starting work
Before starting a work on a pull request double check that no one else is working on the ticket in both issue tickets and pull requests.

### If an issue ticket exists
### If an issue ticket exists
If an issue exists check the ticket to ensure no one else has started work. If first to start work, comment on the ticket to make it evident to others. If the comment looks old or abandoned leave a comment asking if you may start work.

### If an issue ticket doesn't exist
Expand Down Expand Up @@ -122,7 +122,7 @@ install these extra dependencies, Docker can be used. See: [Building documentati

```bash
$ git add modified_files
$ git commit
$ git commit -m "commit message here"
```

to record your changes locally.
Expand Down Expand Up @@ -171,13 +171,13 @@ tools:
* Save plots as part of tests. Plots will save to a directory named test_images by default

```bash
$ pytest arviz/tests/<name of test>.py --save
$ pytest arviz/tests/base_tests/<name of test>.py --save
```

* Optionally save plots to a user named directory. This is useful for comparing changes across branches

```bash
$ pytest arviz/tests/<name of test>.py --save user_defined_directory
$ pytest arviz/tests/base_tests/<name of test>.py --save user_defined_directory
```


Expand All @@ -188,7 +188,7 @@ tools:
$ pytest --cov=arviz --cov-report=html arviz/tests/
```

* Your code has been formatted with [black](https://github.com/ambv/black) with a line length of 100 characters. Note that black only runs in Python 3.6
* Your code has been formatted with [black](https://github.com/ambv/black) with a line length of 100 characters. Note that black only runs in Python 3.6+

```bash
$ pip install black
Expand All @@ -213,6 +213,33 @@ tools:
We have provided a Dockerfile which helps for isolating build problems, and local development.
Install [Docker](https://www.docker.com/) for your operating system, clone this repo. Docker will generate an environment with your local copy of `arviz` with all the packages in Dockerfile.

### container.sh & container.ps1

Predefined docker commands can be run with a `./scripts/container.sh` (on Linux and macOS)
and with `./scripts/container.ps1`. The scripts enables developer easily to call predefined docker commands.
User can use one or multiple flags.

They are executed on the following order: clear-cache, build, test, docs, shell, notebook, lab

$ ./scripts/container.sh --clear-cache
$ ./scripts/container.sh --build

$ ./scripts/container.sh --test
$ ./scripts/container.sh --docs
$ ./scripts/container.sh --shell
$ ./scripts/container.sh --notebook
$ ./scripts/container.sh --lab


$ powershell.exe -File ./scripts/container.ps1 --clear-cache
$ powershell.exe -File ./scripts/container.ps1 --build

$ powershell.exe -File ./scripts/container.ps1 --test
$ powershell.exe -File ./scripts/container.ps1 --docs
$ powershell.exe -File ./scripts/container.ps1 --shell
$ powershell.exe -File ./scripts/container.ps1 --notebook
$ powershell.exe -File ./scripts/container.ps1 --lab

### Testing in Docker
Testing the code using docker consists of executing the same file 3 times (you may need root privileges to run it).
First run `./scripts/container.sh --clear-cache`. Then run `./scripts/container.sh --build`. This starts a local docker image called `arviz`. Finally run the tests with `./scripts/container.sh --test`. This should be quite close to how the tests run on TravisCI.
Expand All @@ -229,17 +256,27 @@ To start a bash shell inside Docker, run:

$ docker run --mount type=bind,source="$(pwd)",target=/opt/arviz/ -it arviz bash

and for Windows, use %CD% on cmd.exe and $pwd.Path on powershell.

$ docker run --mount type=bind,source=%CD%,target=/opt/arviz/ -it arviz bash

Alternatively, to start a jupyter notebook, there are two steps, first run:

$ docker run --mount type=bind,source="$(pwd)",target=/opt/arviz/ --name jupyter-dock -it -d -p 8888:8888 arviz
$ docker exec jupyter-dock bash -c "pip install jupyter"
$ docker exec -it jupyter-dock bash -c "jupyter notebook --ip 0.0.0.0 --no-browser --allow-root"

and the same on Windows

$ docker run --mount type=bind,source=%CD%,target=/opt/arviz/ --name jupyter-dock -it -d -p 8888:8888 arviz
$ docker exec jupyter-dock bash -c "pip install jupyter"
$ docker exec -it jupyter-dock bash -c "jupyter notebook --ip 0.0.0.0 --no-browser --allow-root"

This will output something similar to `http://(<docker container id> or <ip>):8888/?token=<token id>`, and can be accessed at `http://localhost:8888/?token=<token id>`.

### Building documentation with Docker
The documentation can be build with Docker by running `./scripts/container.sh
--sphinx-build`. The docker image contains by default all dependencies needed
--docs`. The docker image contains by default all dependencies needed
for building the documentation. After having build the docs in the Docker
container, they can be checked at `doc/build`.

Expand Down
2 changes: 1 addition & 1 deletion examples/bokeh/bokeh_plot_pair_point_estimate.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"""
Point Estimate Pairplot
===============
=======================
_thumb: .2, .5
"""
Expand Down
2 changes: 1 addition & 1 deletion examples/matplotlib/mpl_plot_pair_point_estimate.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"""
Point Estimate Pairplot
===============
=======================
_thumb: .2, .5
"""
Expand Down
22 changes: 10 additions & 12 deletions scripts/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,6 @@ ARG PYTORCH_VERSION
ARG EMCEE_VERSION
ARG TF_VERSION
ARG PYMC3_VERSION
ARG COVERALLS_PARALLEL
ARG NAME

ENV PYTHON_VERSION=${PYTHON_VERSION}
ENV PYSTAN_VERSION=${PYSTAN_VERSION}
Expand All @@ -20,8 +18,6 @@ ENV PYTORCH_VERSION=${PYTORCH_VERSION}
ENV EMCEE_VERSION=${EMCEE_VERSION}
ENV TF_VERSION=${TF_VERSION}
ENV PYMC3_VERSION=${PYMC3_VERSION}
ENV COVERALLS_PARALLEL=${COVERALLS_PARALLEL}
ENV NAME=${NAME}

# Change behavior of create_test.sh script
ENV DOCKER_BUILD=true
Expand All @@ -32,29 +28,31 @@ ENV LANG=C.UTF-8

# Update container
RUN apt-get update && apt-get install -y git build-essential pandoc vim ffmpeg \
&& rm -rf /var/lib/apt/lists/*
dos2unix libgtk-3-dev libdbus-glib-1-dev libxt-dev && rm -rf /var/lib/apt/lists/*


# Copy requirements and environment installation scripts
COPY $SRC_DIR/requirements.txt opt/arviz/
COPY $SRC_DIR/requirements-dev.txt opt/arviz/
COPY $SRC_DIR/scripts/ opt/arviz/scripts
COPY $SRC_DIR/requirements.txt /opt/arviz/
COPY $SRC_DIR/requirements-dev.txt /opt/arviz/
COPY $SRC_DIR/requirements-docs.txt /opt/arviz/
COPY $SRC_DIR/requirements-external.txt /opt/arviz/
COPY $SRC_DIR/requirements-optional.txt /opt/arviz/
COPY $SRC_DIR/scripts/ /opt/arviz/scripts
RUN find /opt/arviz/ -type f -print0 | xargs -0 dos2unix
WORKDIR /opt/arviz


# Create conda environment. Defaults to Python 3.6
# Create conda environment. Defaults to Python 3.7
RUN ./scripts/create_testenv.sh


# Set automatic conda activation in non interactive and shells
ENV BASH_ENV="/root/activate_conda.sh"
RUN echo ". /root/activate_conda.sh" > /root/.bashrc


# Remove conda cache
RUN conda clean --all

COPY $SRC_DIR /opt/arviz
RUN find /opt/arviz/ -type f -print0 | xargs -0 dos2unix

# Clear any cached models from copied from host filesystem
RUN rm -f arviz/tests/saved_models/*.pkl
Expand Down
86 changes: 86 additions & 0 deletions scripts/container.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
if (!(Test-Path env:SRC_DIR)) {
$SRC_DIR=$pwd.Path
}

$build = $false
$clearCache = $false
$docs = $false
$lab = $false
$notebook = $false
$shell = $false
$test = $false

foreach ($arg in $args) {
if ($arg -eq "--build") {
$build = $true
}
elseif ($arg -eq "--clear-cache") {
$clearCache = $true
}
elseif ($arg -eq "--docs") {
$docs = $true
}
elseif ($arg -eq "--lab") {
$lab = $true
}
elseif ($arg -eq "--notebook") {
$notebook = $true
}
elseif ($arg -eq "--shell") {
$shell = $true
}
elseif ($arg -eq "--test") {
$test = $true
}
}

if ($build) {
echo "Building Docker Image"
# Build container for use of testing or notebook
docker build \
-t arviz \
-f $SRC_DIR/scripts/Dockerfile `
--build-arg SRC_DIR=. $SRC_DIR `
--build-arg PYTHON_VERSION=${PYTHON_VERSION} `
--build-arg PYSTAN_VERSION=${PYSTAN_VERSION} `
--build-arg PYRO_VERSION=${PYRO_VERSION} `
--build-arg PYTORCH_VERSION=${PYTORCH_VERSION} `
--build-arg EMCEE_VERSION=${EMCEE_VERSION} `
--build-arg TF_VERSION=${TF_VERSION} `
--build-arg PYMC3_VERSION=${PYMC3_VERSION}
}

if ($clearCache) {
echo "Removing cached files and models"
Get-ChildItem -Path $SRC_DIR -Recurse -Force -Include "__pycache__" | Remove-Item -Force -Recurse
Get-ChildItem -Path ($SRC_DIR + "\arviz\tests\saved_models") -Recurse -Force -Include "*.pkl" | Remove-Item -Force -Recurse
}

if ($test) {
echo "Testing Arviz"
docker run --mount type=bind,source=$SRC_DIR,target=/opt/arviz/ --rm arviz:latest bash -c `
"NUMBA_DISABLE_JIT=1 pytest -v arviz/tests/ --cov=arviz/"
}

if ($docs) {
echo "Build docs with sphinx"
docker run --mount type=bind,source=$SRC_DIR,target=/opt/arviz --name arviz_sphinx --rm arviz:latest bash -c `
"if [ -d ./doc/build ]; then python -msphinx -M clean doc doc/build; fi && sphinx-build doc doc/build -b html"
}

if ($shell) {
echo "Starting Arviz Container Shell"
docker run -it --mount type=bind,source=$SRC_DIR,target=/opt/arviz --name arviz_shell --rm arviz:latest /bin/bash
}

if ($notebook) {
echo "Starting Arviz Container Jupyter server"
docker run --mount type=bind,source=$SRC_DIR,target=/opt/arviz/ --name jupyter-dock -it -d -p 8888:8888 arviz
docker exec -it jupyter-dock bash -c "jupyter notebook --ip 0.0.0.0 --no-browser --allow-root"
}

if ($lab) {
echo "Starting Arviz Container Jupyter server with Jupyter Lab interface"
docker run --mount type=bind,source=$SRC_DIR,target=/opt/arviz/ --name jupyter-dock -it -d -p 8888:8888 arviz
docker exec -it jupyter-dock bash -c "jupyter lab --ip 0.0.0.0 --no-browser --allow-root"
}
42 changes: 27 additions & 15 deletions scripts/container.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
#! /bin/bash
SRC_DIR=${SRC_DIR:-`pwd`}
NAME=${NAME:-SPHINX}

if [[ $* == *--clear-cache* ]]; then
echo "Removing cached files and models"
find -type d -name __pycache__ -exec rm -rf {} +
rm -f arviz/tests/saved_models/*.pkl
fi

# Build container for use of testing or notebook
if [[ $* == *--build* ]]; then
Expand All @@ -15,15 +20,21 @@ if [[ $* == *--build* ]]; then
--build-arg PYTORCH_VERSION=${PYTORCH_VERSION} \
--build-arg EMCEE_VERSION=${EMCEE_VERSION} \
--build-arg TF_VERSION=${TF_VERSION} \
--build-arg PYMC3_VERSION=${PYMC3_VERSION} \
--build-arg NAME=${NAME}
--build-arg PYMC3_VERSION=${PYMC3_VERSION}
fi


if [[ $* == *--clear-cache* ]]; then
echo "Removing cached files and models"
find -type d -name __pycache__ -exec rm -rf {} +
rm -f arviz/tests/saved_models/*.pkl
if [[ $* == *--test* ]]; then
echo "Testing Arviz"
docker run --mount type=bind,source="$(pwd)",target=/opt/arviz/ --rm arviz:latest bash -c \
"NUMBA_DISABLE_JIT=1 pytest -v arviz/tests/ --cov=arviz/"
fi


if [[ $* == *--docs* ]]; then
echo "Build docs with sphinx"
docker run --mount type=bind,source="$(pwd)",target=/opt/arviz --name arviz_sphinx --rm arviz:latest bash -c \
"if [ -d ./doc/build ]; then python -msphinx -M clean doc doc/build; fi && sphinx-build doc doc/build -b html"
fi


Expand All @@ -32,15 +43,16 @@ if [[ $* == *--shell* ]]; then
docker run -it --mount type=bind,source="$(pwd)",target=/opt/arviz --name arviz_shell --rm arviz:latest /bin/bash
fi

if [[ $* == *--sphinx-build* ]]; then
echo "Build docs with sphinx"
docker run --mount type=bind,source="$(pwd)",target=/opt/arviz --name arviz_sphinx --rm arviz:latest bash -c \
"if [ -d ./doc/build ]; then python -msphinx -M clean doc doc/build; fi && sphinx-build doc doc/build -b html"

if [[ $* == *--notebook* ]]; then
echo "Starting Arviz Container Jupyter server"
docker run --mount type=bind,source="$(pwd)",target=/opt/arviz/ --name jupyter-dock -it -d -p 8888:8888 arviz
docker exec -it jupyter-dock bash -c "jupyter notebook --ip 0.0.0.0 --no-browser --allow-root"
fi


if [[ $* == *--test* ]]; then
echo "Testing Arviz"
docker run --mount type=bind,source="$(pwd)",target=/opt/arviz/ --rm arviz:latest bash -c \
"NUMBA_DISABLE_JIT=1 pytest -v arviz/tests/ --cov=arviz/"
if [[ $* == *--lab* ]]; then
echo "Starting Arviz Container Jupyter server with Jupyter Lab interface"
docker run --mount type=bind,source="$(pwd)",target=/opt/arviz/ --name jupyter-dock -it -d -p 8888:8888 arviz
docker exec -it jupyter-dock bash -c "jupyter lab --ip 0.0.0.0 --no-browser --allow-root"
fi
17 changes: 10 additions & 7 deletions scripts/create_testenv.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,14 @@ command -v conda >/dev/null 2>&1 || {
exit 1;
}

# if no python specified, use Travis version, or else 3.6
PYTHON_VERSION=${PYTHON_VERSION:-${TRAVIS_PYTHON_VERSION:-3.6}}
# if no python specified, use Travis version, or else 3.7
PYTHON_VERSION=${PYTHON_VERSION:-${TRAVIS_PYTHON_VERSION:-3.7}}
PYSTAN_VERSION=${PYSTAN_VERSION:-latest}
PYTORCH_VERSION=${PYTORCH_VERSION:-latest}
PYRO_VERSION=${PYRO_VERSION:-latest}
EMCEE_VERSION=${EMCEE_VERSION:-latest}
TF_VERSION=${TF_VERSION:-latest}
PYMC3_VERSION=${PYMC3_VERSION:-latest}
NAME=${NAME:-UNIT}


if [[ $* != *--global* ]]; then
Expand Down Expand Up @@ -93,9 +92,13 @@ fi


# Install editable using the setup.py
pip install --no-cache-dir -r requirements.txt
pip install --no-cache-dir -r requirements.txt
pip install --no-cache-dir -r requirements-dev.txt
pip install --no-cache-dir -r requirements-docs.txt
pip install --no-cache-dir -r requirements-external.txt
pip install --no-cache-dir -r requirements-optional.txt

if [ "$NAME" = "SPHINX" ]; then
conda install -y -c conda-forge selenium phantomjs
fi
conda install -y geckodriver firefox jupyterlab ipywidgets nodejs --channel conda-forge

jupyter nbextension enable --py widgetsnbextension
jupyter labextension install @jupyter-widgets/jupyterlab-manager
8 changes: 0 additions & 8 deletions scripts/deploy_docs.sh

This file was deleted.

4 changes: 2 additions & 2 deletions scripts/lint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ python -m pydocstyle --convention=numpy ${SRC_DIR}/arviz/
echo "Success!"

echo "Checking code style with black..."
python -m black -l 100 --check arviz/ examples/
python -m black -l 100 --check ${SRC_DIR}/arviz/ ${SRC_DIR}/examples/
echo "Success!"

echo "Checking code style with pylint..."
python -m pylint arviz/
python -m pylint ${SRC_DIR}/arviz/
echo "Success!"

0 comments on commit 3352a8a

Please sign in to comment.