Skip to content

dask/dask-docker

Folders and files

NameName
Last commit message
Last commit date

Latest commit

62fa18b · Jan 8, 2025
Jan 7, 2025
Jan 8, 2025
Jan 7, 2025
Jan 8, 2025
Oct 24, 2019
Oct 21, 2019
Dec 10, 2019
Jan 7, 2025
Mar 2, 2022

Repository files navigation

Dask docker images

Docker build

Image Description Versions
ghcr.io/dask/dask Base image to use for Dask scheduler and workers

ghcr.io/dask/dask-notebook Jupyter Notebook image to use as helper entrypoint

Example

An example docker-compose.yml file is included for starting a small cluster.

docker-compose up

Open the notebook using the URL that is printed by the output so it has the token.

On a new notebook run:

from dask.distributed import Client
client = Client()  # The address is automatically set by the DASK_SCHEDULER_ADDRESS environment variable
client.ncores()

It should output something like this:

{'tcp://172.23.0.4:41269': 4}

Environment Variables

The following environment variables are supported for both the base and notebook images:

  • $EXTRA_APT_PACKAGES - Space separated list of additional system packages to install with apt.
  • $EXTRA_CONDA_PACKAGES - Space separated list of additional packages to install with conda. This variable can also be used to specify custom conda channels; for example, to install the latest Dask conda nightly packages:
docker run -e EXTRA_CONDA_PACKAGES="-c dask/label/dev dask" daskdev/dask:latest
  • $EXTRA_PIP_PACKAGES - Space separated list of additional python packages to install with pip.
  • $USE_MAMBA - Boolean controlling whether to use conda or mamba to install $EXTRA_CONDA_PACKAGES.

The notebook image supports the following additional environment variables:

  • $JUPYTERLAB_ARGS - Extra arguments to pass to the jupyter lab command.

Building images

Docker compose provides an easy way to building all the images with the right context

cd build

# Use legacy builder as buildkit still doesn't support subdirectories when building from git repos
export DOCKER_BUILDKIT=0
export COMPOSE_DOCKER_CLI_BUILD=0

docker-compose build

# Just build one image e.g. notebook
docker-compose build notebook

Cross building

The images can be cross-built using docker buildx bake. However buildx bake does not listen to depends_on (since in theory that is only a runtime not a build time constraint docker/buildx#447). To work around this we first need to build the "docker-stacks-foundation" image.

cd build

# If you have permission to push to daskdev/
docker buildx bake --progress=plain --set *.platform=linux/arm64,linux/amd64 --push docker-stacks-foundation
docker buildx bake --progress=plain --set *.platform=linux/arm64,linux/amd64 --push

# If you don'tset DOCKERUSER to your dockerhub username.
export DOCKERUSER=holdenk
docker buildx bake --progress=plain --set *.platform=linux/arm64,linux/amd64 --set docker-stacks-foundation.tags.image=${DOCKERUSER}/docker-stacks-foundation:lab-py38 --push docker-stacks-foundation
docker buildx bake --progress=plain --set *.platform=linux/arm64,linux/amd64 --set scheduler.tags=${DOCKERUSER}/dask --set worker.tags=${DOCKERUSER}/dask --set notebook.tags=${DOCKERUSER}/dask-notebook --set docker-stacks-foundation.tags=${DOCKERUSER}/docker-stacks-foundation:lab-py38 --set notebook.args.base=${DOCKERUSER} --push

Releasing

Building and releasing new image versions is done automatically.

  • When a new Dask version is released the watch-conda-forge action will trigger and open a PR to update the latest release version in this repo.
  • If images build successfully that PR will be automatically merged by the automerge action.
  • When a PR like this is merged which updates the pinned release version a tag is automatically created to match that version by the autotag action.
  • When tags are created a new image is built and pushed using the docker/build-push-action action.