[RFC] Bakefile Multi Arch builds

[ryanolee]

Note

This is a request for comment, it is not a final implementation. I am happy to push it forwards to a completed state if it seems like it will be useful to the overall project structure / aims. Though given how substantial this change is to the overall build process can understand if the changes are too large for the stated aims of this PR.

Warning

It should be noted that bakefiles are still somewhat experimental https://docs.docker.com/build/bake/

First thanks all for the amazing project, really cool to try out this project!

What

This PR aims to address #290 #330 and #128 by migrating the build process from github actions into a docker bakefile for more centralized control over which docker files are built and to allow for multi-arch builds.

Why

In order to be able to build multiarch docker files so that ARM platforms can be tested with. Especially with ARM containers normally being cheaper to run on certain CaaS Platforms for similar amounts of compute. (Also I thought it would be funny to run dask on a raspberry Pi cluster🙊

How

This PR implements a new bakefile which aims to simplify the overall build flow in CI offloading much of the responsibility to docker itself.

It should be noted that:

• Tagging as been replicated in the bakefile as it currently functions in CI
• Base images are pulled directly from GIT to be built at the versions we require things to be built with

How to test

See passing CI Jobs here: https://github.com/ryanolee/dask-docker/actions
And relevant docker images here: https://github.com/ryanolee?tab=packages&repo_name=dask-docker

Try:

version: "3.1"

services:
  scheduler:
    image: ghcr.io/ryanolee/dask:2024.9.1-py3.12
    hostname: scheduler
    ports:
      - "8786:8786"
      - "8787:8787"
    command: ["dask-scheduler"]

  worker:
    image: ghcr.io/ryanolee/dask:2024.9.1-py3.12
    command: ["dask-worker", "tcp://scheduler:8786"]
    # For Docker swarm you can specify multiple workers, this is ignored by `docker-compose up`
    deploy:
      replicas: 4

  notebook:
    image: ghcr.io/ryanolee/dask-notebook:2024.9.1-py3.12
    ports:
      - "8888:8888"
    environment:
      - DASK_SCHEDULER_ADDRESS="tcp://scheduler:8786"

OR

Run make build-docker and import the images from your build context and reference them in a similar way.

Has been tested on 4 Node Raspberry Pi cluster using Docker swarm and works as expected for everything I have tried it with so for.

x86 Builds have been tested on WSL2, Ubuntu 24.04 and have worked as expected also.