New aiida docker stack with a slim base image

[unkcpz]

New aiida docker stack based on a slim image

Background and problems description

Thanks to the fantastic work of @csadorf, we have aiidalab-docker-stack held for four image variants:

• base – A minimal image that comes with AiiDA pre-installed and an AiiDA profile set up.
• base-with-services – Like base, but AiiDA services (PostgreSQL and RabbitMQ) are installed on the container and automatically launched on startup.
• lab – Like base, but uses the AiiDAlab home app as the primary interface (the standard JupyterLab interface is also available).
• full-stack – Our most comprehensive image, like lab, but also comes with services pre-installed and launched.

With this new way of maintaining aiida/aiidalab images, the aiida-prerequisites is not used anymore. We need to find a way to elegantly replace the old approach and make the new approach more automatic and easy to maintain.

System user name can not be changed

A knew issue from using jupyter/minimal-notebook as the base image, the system user name default set to jovyan and can only be changed by running container as root. If the container starts as a root user, it is hard to set start-up scripts to configure AiiDA for the system user of the container. Meanwhile, login to the container will end up with root user will bringing the complexity for vscode attach container setup.

Not able to keep the latest aiida-core version

When the new aiida-core version is released or the new fixes merge to main, users may want to immediately use the image with newly added features. Now, it has to wait until aiidalab-docker-stack bump the version of aiida-core, then builds and uploads the images.

Confuse the organization name of docker hub

Although the docker hub namespaces aiidateam and aiidalab are all managed by us, we still try to not confuse and mix up the images with these two different namespaces.

Goals

Archive aiidateam/aiida-prerequisites and have all aiida only related docker image build files maintained in aiida-core.

Two docker images, aiidateam/base and aiiidateam/base-with-services

• Having a docker image mainly containing just aiida-core installed and can be started with rabbitmq & PostgreSQL services image by docker-compose.
• Having a docker image based on the base image above but having rabbitmq & Postgresql installed and configure inside the container.

Images support both linux/amd64 and linux/arm64, and include basic system tools: vim, git, conda/mamba for quick development environment buildup.

The main purposes of these images included but were not limited to:

• As a base image for the aiidalab docker stack that is able to be deployed to the cloud for tutorial or testing, refer to aiidalab-on-azure.
• For quick and clean development environment build and launch. With the current aiidalab-launch user can quickly start a workable container with tools and aiida environment fully set up and start plugin or aiidalab app development immediately.

Proposed enhancement

Using mambaorg/micromamba as base image

Actively maintained by the mamba team (https://github.com/mamba-org/mamba#micromamba). The mambaorg/micromamba image is a lightweight and fast image for fast building of small conda-based containers. It small but versatile image provides a flexible way to adjust the container as we want. The image supports linux/amd64 and linux/arm64 with running ubuntu as the base OS.
The image allows running commands in Dockerfile through root and system user under a specific conda environment by using a simple entrypoint script as the prefix. It makes configuring AiiDA environment as a system user running in a conda environment possible.

I test this idea on aiidalab/aiidalab-docker-stack#319 and all works.

Move the base and base-with-services stacks into aiida-core repo

Create docker files and build based on mambaorg/micromamba, the images are built and upload to aiidateam docker hub namespace. The docker files are maintained in the aiida-core repository with github action used to build and upload the images.
After having these images, the aiidalab docker stack can then build on them and reduce the duplication but focus on providing the jupyter tools.

Having self-hosted runner for arm64 CI test

GitHub does not provided free CI server for running CI workflows on architecture other than the x86_64. But they allow users to easily build CI runner in a self-host way. Once there is a server running ARM64 architecture, the CI tests for build and test image can be run from the self-hosted runner.

[ltalirz]

I like having the two variants of the images (with and without services).

To me, the main question here would be whether the AiiDAlab team can take the base-with-services container to add their AiiDAlab stuff on top or whether they are better served by starting from one of the containers provided by project Jupyter.