py3: Docker images for python3-based Sage

[sagetrac-gitlab-bot]

Docker images for python3-based Sage are useful in general,
and in particular for continuous integration or other testing
workflows for packages that depend on Sage.

The infrastructure introduced by #24655 automatically produces
Docker images for python2-based Sage. In order to extend it to
also produce python3-based ones,

• we update the micro_release target for make in Makefile
  to not remove the config directory

• we adapt .gitlab-ci.yml and docker/Dockerfile,
  duplicating all the docker targets for py3, creating
  a WITH_PYTHON and a DOCKER_TAG_SUFFIX
  environment variables, and using them in the shell
  scripts docker/*.sh.

To play with the generated Python 2/3 docker images, you can go to https://gitlab-hooks-flau3jeaza-ew.a.run.app/status/trac/branch/u%2Fsaraedum%2F28041

CC: @saraedum @zerline @roed314

Component: distribution

Keywords: gitlab-ci, days101

Author: Samuel Lelièvre, Julian Rüth

Branch/Commit: ca24332

Reviewer: David Roe

Issue created by migration from https://trac.sagemath.org/ticket/28041

[slel]

Changed keywords from none to gitlab-ci

[slel]

comment:2

Still need to make the changes to the docker/*.sh shell scripts.

[slel]

Changed keywords from gitlab-ci to gitlab-ci, days101

[embray]

comment:4

I'm trying to get this working, but the build is stalling out at:

Step 61/80 : RUN sudo $SAGE_ROOT/sage --nodotsage -c "install_scripts('/usr/bin')"
 ---> Running in d4ce02ca02d6
Forcing sage-location, probably because a new package was installed.
Cleaning up, do not interrupt this.
Done cleaning.
Traceback (most recent call last):
  File "/home/sage/sage/src/bin/sage-eval", line 4, in <module>
    from sage.all import *
ImportError: No module named sage.all
The command '/bin/sh -c sudo $SAGE_ROOT/sage --nodotsage -c "install_scripts('/usr/bin')"' returned a non-zero code: 1

The problem is that SAGE_PYTHON_VERSION is not being set properly when running $SAGE_ROOT/sage.

This is because there are two copies of sage-env-config, where SAGE_PYTHON_VERSION is defined: One under $SAGE_ROOT/src/bin and one other $SAGE_ROOT/local/bin. This is normal when doing Sage development, and I believe, if it exists, it's the one under src/bin that will be used by default when developing/building.

The problem in this case is that although the one in local/bin correctly has SAGE_PYTHON_VERSION=3, the one in src/bin still has SAGE_PYTHON_VERSION=2 for some reason.

I wonder if this is a remnant from an earlier build stage that was cached, but not properly updated for some reason (I thought that different --build-arg flags would avoid use incompatible cached images built with different --build-args, but it's possible that that arg isn't being included in the build early enough.

[embray]

comment:5

The problem seems to start here:

Step 56/80 : RUN make micro_release
 ---> Using cache
 ---> 2fc1a353e2ac

When running make micro_release, ./configure gets re-run, but without any flags we passed it previously. So it doesn't preserve our --with-python=3 from earlier.

[embray]

comment:6

(CC Odile who was interested in having a working Python 3 Docker image)

[saraedum]

comment:7

Sorry for this. We had already figured the broken ./configure when micro_release is run. I thought that Samuel had pushed a fix for this here already.

[saraedum]

comment:8

Isn't the preservation of config good enough for this? (See the changeset here.)

[embray]

comment:9

I was just about to say: I think the preservation of config.status is good enough in fact.

[embray]

comment:10

Specifically, one of the prereqs of micro_release is the bdist-clean target, which in turn calls $(MAKE) misc-clean which deletes config.status, forcing configure to be re-run.

[saraedum]

comment:11

What the bdist-clean? Is this a automake target? Otherwise is it reasonable that the build system's config is visible after running it?

[embray]

comment:12

Not sure, but I think just swapping sagelib-clean and bdist-clean in the prereqs for micro_release should suffice. It's fragile, but I think it will work. Testing now.

[embray]

comment:13

Replying to @saraedum:

What the bdist-clean? Is this a automake target? Otherwise is it reasonable that the build system's config is visible after running it?

No, this is just a pseudo-target in Sage's top-level Makefile. I have no idea why specifically it's named that. It's also used by the fast-rebuild-clean target. It just calls clean (which is in build/make/Makefile for some reason), and then misc-clean.

All clean does apparently is:

clean:
    @echo "Deleting package build directories..."
    rm -rf "$(SAGE_LOCAL)/var/tmp/sage/build"

which is certainly worth doing. But maybe this needs to be detangled a bit...

[slel]

comment:14

The following files in docker/ probably still need changes:

• build-docker.sh
• pull-gitlab.sh
• push-dockerhub.sh
• push-gitlab.sh
• setup-make-parallelity.sh
• test-dev.sh
• update-env.sh

[embray]

comment:15

Yes, most likely. One I just get the docker build working locally I will try adding all the other relevant updates.

It would be nice for starters to get a Python 3 image on Docker Hub, even if I have to push it manually, then get future ones building automatically.

[saraedum]

comment:16

I think that once your Dockerfile goes through, you just need to add your branch to build-from-clean, pull your image from the GitLab registry and push it to Docker Hub.

[embray]

comment:17

For Odile's use I went ahead and pushed a prototype of this up to Dockerhub: sagemath/sagemath-dev-py3:alpha

Unfortunately it still has a bug. The sage-entrypoint still tries to run make build (understandably) which results in re-configuring (needlessly) and breaking things.

A workaround for now is to set the PREREQ_OPTIONS variable which is used by Sage's top-level Makefile when it runs configure. So you can set this when running the container like:

docker run -ti --rm -e PREREQ_OPTIONS='--with-python=3' sagemath/sagemath-dev-py3:alpha

Next up, I think we should modify the Docker build to at least keep config.status so that this doesn't keep happening.

[embray]

comment:18

Apparently I was not able to successfully upload the Docker image to hub.docker.com because (AFAICT) of this issue.

But I was able to upload it to our image registry on gitlab.com. So try:

docker run -ti --rm -e PREREQ_OPTIONS='--with-python=3' registry.gitlab.com/sagemath/dev/sage/sagemath-dev-py3:alpha

You can also use this image as a base for another Docker image by doing something like:

FROM registry.gitlab.com/sagemath/dev/sage/sagemath-dev-py3:alpha
ENV PREREQ_OPTIONS='--with-python=3'
...

That is, you would ensure PREREQ_OPTIONS is set as an environment variable in the new image, by default.

This is not ideal and should be fixed, but it's a start anyways...

[saraedum]

Changed author from Samuel Lelièvre to Samuel Lelièvre, Julian Rüth

[sagetrac-git]

Commit: 89c78f8

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

89c78f8

Cleanup py-3 script

[saraedum]

comment:47

the docker images should be available on binder shortly: https://gitlab.com/sagemath/dev/trac/pipelines/81996545

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

1d02785

Add a test-hook for build-from-clean

[sagetrac-git]

Changed commit from 89c78f8 to 1d02785

[saraedum]

comment:50

I started a full build-from-clean here: https://gitlab.com/sagemath/dev/trac/pipelines/82005383

[sagetrac-git]

Changed commit from 1d02785 to ca24332

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

ca24332

Do not run build-from-latest for explicit builds

[saraedum]

comment:52

The Python 2 and Python 3 images seem to be functional. (At least they work in Binder.)

[roed314]

Reviewer: David Roe

[roed314]

comment:53

Looks good to me. Both Python 2 and Python 3 built fine.

[saraedum]

comment:54

Not sure if this comes from the changes here, but there's a MAKEFLAGS=w -j -l7.8 --jobserver-fds=3,4 V=1 in the output. Let me try to see who put the w there.

[saraedum]

comment:55

Ok. That's how it's supposed to work https://lists.gnu.org/archive/html/help-make/2016-03/msg00009.html.

[fchapoton]

comment:56

moving milestone to 9.0 (after release of 8.9)

[vbraun]

Changed branch from u/saraedum/28041 to ca24332