diff --git a/CHANGELOG.md b/CHANGELOG.md index 5f8d4b3810..ebf6822400 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,7 @@ * Fixed hist kind of `plot_dist` with multidimensional input (#1115) * Fixed `TypeError` in `transform` argument of `plot_density` and `plot_forest` when `InferenceData` is a list or tuple (#1121) * Fixed overlaid pairplots issue (#1135) +* Update Docker building steps (#1127) ### Deprecation ### Documentation diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 69a113806a..d76d1eb76a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -31,7 +31,7 @@ to ensure that any new contribution does not strongly overlap with existing func ## Steps before starting work Before starting a work on a pull request double check that no one else is working on the ticket in both issue tickets and pull requests. -### If an issue ticket exists +### If an issue ticket exists If an issue exists check the ticket to ensure no one else has started work. If first to start work, comment on the ticket to make it evident to others. If the comment looks old or abandoned leave a comment asking if you may start work. ### If an issue ticket doesn't exist @@ -122,7 +122,7 @@ install these extra dependencies, Docker can be used. See: [Building documentati ```bash $ git add modified_files - $ git commit + $ git commit -m "commit message here" ``` to record your changes locally. @@ -171,13 +171,13 @@ tools: * Save plots as part of tests. Plots will save to a directory named test_images by default ```bash - $ pytest arviz/tests/.py --save + $ pytest arviz/tests/base_tests/.py --save ``` * Optionally save plots to a user named directory. This is useful for comparing changes across branches ```bash - $ pytest arviz/tests/.py --save user_defined_directory + $ pytest arviz/tests/base_tests/.py --save user_defined_directory ``` @@ -188,7 +188,7 @@ tools: $ pytest --cov=arviz --cov-report=html arviz/tests/ ``` -* Your code has been formatted with [black](https://github.com/ambv/black) with a line length of 100 characters. Note that black only runs in Python 3.6 +* Your code has been formatted with [black](https://github.com/ambv/black) with a line length of 100 characters. Note that black only runs in Python 3.6+ ```bash $ pip install black @@ -213,6 +213,33 @@ tools: We have provided a Dockerfile which helps for isolating build problems, and local development. Install [Docker](https://www.docker.com/) for your operating system, clone this repo. Docker will generate an environment with your local copy of `arviz` with all the packages in Dockerfile. +### container.sh & container.ps1 + +Predefined docker commands can be run with a `./scripts/container.sh` (on Linux and macOS) +and with `./scripts/container.ps1`. The scripts enables developer easily to call predefined docker commands. +User can use one or multiple flags. + +They are executed on the following order: clear-cache, build, test, docs, shell, notebook, lab + + $ ./scripts/container.sh --clear-cache + $ ./scripts/container.sh --build + + $ ./scripts/container.sh --test + $ ./scripts/container.sh --docs + $ ./scripts/container.sh --shell + $ ./scripts/container.sh --notebook + $ ./scripts/container.sh --lab + + + $ powershell.exe -File ./scripts/container.ps1 --clear-cache + $ powershell.exe -File ./scripts/container.ps1 --build + + $ powershell.exe -File ./scripts/container.ps1 --test + $ powershell.exe -File ./scripts/container.ps1 --docs + $ powershell.exe -File ./scripts/container.ps1 --shell + $ powershell.exe -File ./scripts/container.ps1 --notebook + $ powershell.exe -File ./scripts/container.ps1 --lab + ### Testing in Docker Testing the code using docker consists of executing the same file 3 times (you may need root privileges to run it). First run `./scripts/container.sh --clear-cache`. Then run `./scripts/container.sh --build`. This starts a local docker image called `arviz`. Finally run the tests with `./scripts/container.sh --test`. This should be quite close to how the tests run on TravisCI. @@ -229,17 +256,27 @@ To start a bash shell inside Docker, run: $ docker run --mount type=bind,source="$(pwd)",target=/opt/arviz/ -it arviz bash +and for Windows, use %CD% on cmd.exe and $pwd.Path on powershell. + + $ docker run --mount type=bind,source=%CD%,target=/opt/arviz/ -it arviz bash + Alternatively, to start a jupyter notebook, there are two steps, first run: $ docker run --mount type=bind,source="$(pwd)",target=/opt/arviz/ --name jupyter-dock -it -d -p 8888:8888 arviz $ docker exec jupyter-dock bash -c "pip install jupyter" $ docker exec -it jupyter-dock bash -c "jupyter notebook --ip 0.0.0.0 --no-browser --allow-root" +and the same on Windows + + $ docker run --mount type=bind,source=%CD%,target=/opt/arviz/ --name jupyter-dock -it -d -p 8888:8888 arviz + $ docker exec jupyter-dock bash -c "pip install jupyter" + $ docker exec -it jupyter-dock bash -c "jupyter notebook --ip 0.0.0.0 --no-browser --allow-root" + This will output something similar to `http://( or ):8888/?token=`, and can be accessed at `http://localhost:8888/?token=`. ### Building documentation with Docker The documentation can be build with Docker by running `./scripts/container.sh ---sphinx-build`. The docker image contains by default all dependencies needed +--docs`. The docker image contains by default all dependencies needed for building the documentation. After having build the docs in the Docker container, they can be checked at `doc/build`. diff --git a/examples/bokeh/bokeh_plot_pair_point_estimate.py b/examples/bokeh/bokeh_plot_pair_point_estimate.py index 263d6e14ca..5c55e72d46 100644 --- a/examples/bokeh/bokeh_plot_pair_point_estimate.py +++ b/examples/bokeh/bokeh_plot_pair_point_estimate.py @@ -1,6 +1,6 @@ """ Point Estimate Pairplot -=============== +======================= _thumb: .2, .5 """ diff --git a/examples/matplotlib/mpl_plot_pair_point_estimate.py b/examples/matplotlib/mpl_plot_pair_point_estimate.py index 997bb59416..6aed997b1e 100644 --- a/examples/matplotlib/mpl_plot_pair_point_estimate.py +++ b/examples/matplotlib/mpl_plot_pair_point_estimate.py @@ -1,6 +1,6 @@ """ Point Estimate Pairplot -=============== +======================= _thumb: .2, .5 """ diff --git a/scripts/Dockerfile b/scripts/Dockerfile index 45fb50ab81..a2fd407269 100644 --- a/scripts/Dockerfile +++ b/scripts/Dockerfile @@ -10,8 +10,6 @@ ARG PYTORCH_VERSION ARG EMCEE_VERSION ARG TF_VERSION ARG PYMC3_VERSION -ARG COVERALLS_PARALLEL -ARG NAME ENV PYTHON_VERSION=${PYTHON_VERSION} ENV PYSTAN_VERSION=${PYSTAN_VERSION} @@ -20,8 +18,6 @@ ENV PYTORCH_VERSION=${PYTORCH_VERSION} ENV EMCEE_VERSION=${EMCEE_VERSION} ENV TF_VERSION=${TF_VERSION} ENV PYMC3_VERSION=${PYMC3_VERSION} -ENV COVERALLS_PARALLEL=${COVERALLS_PARALLEL} -ENV NAME=${NAME} # Change behavior of create_test.sh script ENV DOCKER_BUILD=true @@ -32,29 +28,31 @@ ENV LANG=C.UTF-8 # Update container RUN apt-get update && apt-get install -y git build-essential pandoc vim ffmpeg \ - && rm -rf /var/lib/apt/lists/* + dos2unix libgtk-3-dev libdbus-glib-1-dev libxt-dev && rm -rf /var/lib/apt/lists/* # Copy requirements and environment installation scripts -COPY $SRC_DIR/requirements.txt opt/arviz/ -COPY $SRC_DIR/requirements-dev.txt opt/arviz/ -COPY $SRC_DIR/scripts/ opt/arviz/scripts +COPY $SRC_DIR/requirements.txt /opt/arviz/ +COPY $SRC_DIR/requirements-dev.txt /opt/arviz/ +COPY $SRC_DIR/requirements-docs.txt /opt/arviz/ +COPY $SRC_DIR/requirements-external.txt /opt/arviz/ +COPY $SRC_DIR/requirements-optional.txt /opt/arviz/ +COPY $SRC_DIR/scripts/ /opt/arviz/scripts +RUN find /opt/arviz/ -type f -print0 | xargs -0 dos2unix WORKDIR /opt/arviz - -# Create conda environment. Defaults to Python 3.6 +# Create conda environment. Defaults to Python 3.7 RUN ./scripts/create_testenv.sh - # Set automatic conda activation in non interactive and shells ENV BASH_ENV="/root/activate_conda.sh" RUN echo ". /root/activate_conda.sh" > /root/.bashrc - # Remove conda cache RUN conda clean --all COPY $SRC_DIR /opt/arviz +RUN find /opt/arviz/ -type f -print0 | xargs -0 dos2unix # Clear any cached models from copied from host filesystem RUN rm -f arviz/tests/saved_models/*.pkl diff --git a/scripts/container.ps1 b/scripts/container.ps1 new file mode 100644 index 0000000000..81ea5b3add --- /dev/null +++ b/scripts/container.ps1 @@ -0,0 +1,86 @@ +if (!(Test-Path env:SRC_DIR)) { + $SRC_DIR=$pwd.Path +} + +$build = $false +$clearCache = $false +$docs = $false +$lab = $false +$notebook = $false +$shell = $false +$test = $false + +foreach ($arg in $args) { + if ($arg -eq "--build") { + $build = $true + } + elseif ($arg -eq "--clear-cache") { + $clearCache = $true + } + elseif ($arg -eq "--docs") { + $docs = $true + } + elseif ($arg -eq "--lab") { + $lab = $true + } + elseif ($arg -eq "--notebook") { + $notebook = $true + } + elseif ($arg -eq "--shell") { + $shell = $true + } + elseif ($arg -eq "--test") { + $test = $true + } +} + +if ($build) { + echo "Building Docker Image" + # Build container for use of testing or notebook + docker build \ + -t arviz \ + -f $SRC_DIR/scripts/Dockerfile ` + --build-arg SRC_DIR=. $SRC_DIR ` + --build-arg PYTHON_VERSION=${PYTHON_VERSION} ` + --build-arg PYSTAN_VERSION=${PYSTAN_VERSION} ` + --build-arg PYRO_VERSION=${PYRO_VERSION} ` + --build-arg PYTORCH_VERSION=${PYTORCH_VERSION} ` + --build-arg EMCEE_VERSION=${EMCEE_VERSION} ` + --build-arg TF_VERSION=${TF_VERSION} ` + --build-arg PYMC3_VERSION=${PYMC3_VERSION} +} + +if ($clearCache) { + echo "Removing cached files and models" + Get-ChildItem -Path $SRC_DIR -Recurse -Force -Include "__pycache__" | Remove-Item -Force -Recurse + Get-ChildItem -Path ($SRC_DIR + "\arviz\tests\saved_models") -Recurse -Force -Include "*.pkl" | Remove-Item -Force -Recurse +} + +if ($test) { + echo "Testing Arviz" + docker run --mount type=bind,source=$SRC_DIR,target=/opt/arviz/ --rm arviz:latest bash -c ` + "NUMBA_DISABLE_JIT=1 pytest -v arviz/tests/ --cov=arviz/" +} + +if ($docs) { + echo "Build docs with sphinx" + docker run --mount type=bind,source=$SRC_DIR,target=/opt/arviz --name arviz_sphinx --rm arviz:latest bash -c ` + "if [ -d ./doc/build ]; then python -msphinx -M clean doc doc/build; fi && sphinx-build doc doc/build -b html" +} + +if ($shell) { + echo "Starting Arviz Container Shell" + docker run -it --mount type=bind,source=$SRC_DIR,target=/opt/arviz --name arviz_shell --rm arviz:latest /bin/bash +} + +if ($notebook) { + echo "Starting Arviz Container Jupyter server" + docker run --mount type=bind,source=$SRC_DIR,target=/opt/arviz/ --name jupyter-dock -it -d -p 8888:8888 arviz + docker exec -it jupyter-dock bash -c "jupyter notebook --ip 0.0.0.0 --no-browser --allow-root" +} + +if ($lab) { + echo "Starting Arviz Container Jupyter server with Jupyter Lab interface" + docker run --mount type=bind,source=$SRC_DIR,target=/opt/arviz/ --name jupyter-dock -it -d -p 8888:8888 arviz + docker exec -it jupyter-dock bash -c "jupyter lab --ip 0.0.0.0 --no-browser --allow-root" +} diff --git a/scripts/container.sh b/scripts/container.sh index 46b91467ce..766f999aac 100755 --- a/scripts/container.sh +++ b/scripts/container.sh @@ -1,6 +1,11 @@ #! /bin/bash SRC_DIR=${SRC_DIR:-`pwd`} -NAME=${NAME:-SPHINX} + +if [[ $* == *--clear-cache* ]]; then + echo "Removing cached files and models" + find -type d -name __pycache__ -exec rm -rf {} + + rm -f arviz/tests/saved_models/*.pkl +fi # Build container for use of testing or notebook if [[ $* == *--build* ]]; then @@ -15,15 +20,21 @@ if [[ $* == *--build* ]]; then --build-arg PYTORCH_VERSION=${PYTORCH_VERSION} \ --build-arg EMCEE_VERSION=${EMCEE_VERSION} \ --build-arg TF_VERSION=${TF_VERSION} \ - --build-arg PYMC3_VERSION=${PYMC3_VERSION} \ - --build-arg NAME=${NAME} + --build-arg PYMC3_VERSION=${PYMC3_VERSION} fi -if [[ $* == *--clear-cache* ]]; then - echo "Removing cached files and models" - find -type d -name __pycache__ -exec rm -rf {} + - rm -f arviz/tests/saved_models/*.pkl +if [[ $* == *--test* ]]; then + echo "Testing Arviz" + docker run --mount type=bind,source="$(pwd)",target=/opt/arviz/ --rm arviz:latest bash -c \ + "NUMBA_DISABLE_JIT=1 pytest -v arviz/tests/ --cov=arviz/" +fi + + +if [[ $* == *--docs* ]]; then + echo "Build docs with sphinx" + docker run --mount type=bind,source="$(pwd)",target=/opt/arviz --name arviz_sphinx --rm arviz:latest bash -c \ + "if [ -d ./doc/build ]; then python -msphinx -M clean doc doc/build; fi && sphinx-build doc doc/build -b html" fi @@ -32,15 +43,16 @@ if [[ $* == *--shell* ]]; then docker run -it --mount type=bind,source="$(pwd)",target=/opt/arviz --name arviz_shell --rm arviz:latest /bin/bash fi -if [[ $* == *--sphinx-build* ]]; then - echo "Build docs with sphinx" - docker run --mount type=bind,source="$(pwd)",target=/opt/arviz --name arviz_sphinx --rm arviz:latest bash -c \ - "if [ -d ./doc/build ]; then python -msphinx -M clean doc doc/build; fi && sphinx-build doc doc/build -b html" + +if [[ $* == *--notebook* ]]; then + echo "Starting Arviz Container Jupyter server" + docker run --mount type=bind,source="$(pwd)",target=/opt/arviz/ --name jupyter-dock -it -d -p 8888:8888 arviz + docker exec -it jupyter-dock bash -c "jupyter notebook --ip 0.0.0.0 --no-browser --allow-root" fi -if [[ $* == *--test* ]]; then - echo "Testing Arviz" - docker run --mount type=bind,source="$(pwd)",target=/opt/arviz/ --rm arviz:latest bash -c \ - "NUMBA_DISABLE_JIT=1 pytest -v arviz/tests/ --cov=arviz/" +if [[ $* == *--lab* ]]; then + echo "Starting Arviz Container Jupyter server with Jupyter Lab interface" + docker run --mount type=bind,source="$(pwd)",target=/opt/arviz/ --name jupyter-dock -it -d -p 8888:8888 arviz + docker exec -it jupyter-dock bash -c "jupyter lab --ip 0.0.0.0 --no-browser --allow-root" fi diff --git a/scripts/create_testenv.sh b/scripts/create_testenv.sh index 462949cad3..a3816cd15d 100755 --- a/scripts/create_testenv.sh +++ b/scripts/create_testenv.sh @@ -7,15 +7,14 @@ command -v conda >/dev/null 2>&1 || { exit 1; } -# if no python specified, use Travis version, or else 3.6 -PYTHON_VERSION=${PYTHON_VERSION:-${TRAVIS_PYTHON_VERSION:-3.6}} +# if no python specified, use Travis version, or else 3.7 +PYTHON_VERSION=${PYTHON_VERSION:-${TRAVIS_PYTHON_VERSION:-3.7}} PYSTAN_VERSION=${PYSTAN_VERSION:-latest} PYTORCH_VERSION=${PYTORCH_VERSION:-latest} PYRO_VERSION=${PYRO_VERSION:-latest} EMCEE_VERSION=${EMCEE_VERSION:-latest} TF_VERSION=${TF_VERSION:-latest} PYMC3_VERSION=${PYMC3_VERSION:-latest} -NAME=${NAME:-UNIT} if [[ $* != *--global* ]]; then @@ -93,9 +92,13 @@ fi # Install editable using the setup.py -pip install --no-cache-dir -r requirements.txt +pip install --no-cache-dir -r requirements.txt pip install --no-cache-dir -r requirements-dev.txt +pip install --no-cache-dir -r requirements-docs.txt +pip install --no-cache-dir -r requirements-external.txt +pip install --no-cache-dir -r requirements-optional.txt -if [ "$NAME" = "SPHINX" ]; then - conda install -y -c conda-forge selenium phantomjs -fi +conda install -y geckodriver firefox jupyterlab ipywidgets nodejs --channel conda-forge + +jupyter nbextension enable --py widgetsnbextension +jupyter labextension install @jupyter-widgets/jupyterlab-manager diff --git a/scripts/deploy_docs.sh b/scripts/deploy_docs.sh deleted file mode 100755 index 162ad22358..0000000000 --- a/scripts/deploy_docs.sh +++ /dev/null @@ -1,8 +0,0 @@ -#!/usr/bin/env bash - -set -x # fail on first error, don't use -e because the token is a secret. - -if [ "${TRAVIS_BRANCH}" = "master" ] && [ "${TRAVIS_PULL_REQUEST}" = "false" ] -then - ghp-import -pfnr https://"${GH_TOKEN}"@github.com/"${TRAVIS_REPO_SLUG}".git doc/build -fi diff --git a/scripts/lint.sh b/scripts/lint.sh index c23139dcf1..59bcfa2e07 100755 --- a/scripts/lint.sh +++ b/scripts/lint.sh @@ -9,9 +9,9 @@ python -m pydocstyle --convention=numpy ${SRC_DIR}/arviz/ echo "Success!" echo "Checking code style with black..." -python -m black -l 100 --check arviz/ examples/ +python -m black -l 100 --check ${SRC_DIR}/arviz/ ${SRC_DIR}/examples/ echo "Success!" echo "Checking code style with pylint..." -python -m pylint arviz/ +python -m pylint ${SRC_DIR}/arviz/ echo "Success!"