GitLab local registry support

[nrcfieldsa]

Is it currently supported to use a GitLab repository or container registry on a private network with shpc? Is it possible with a config to update the base repository url?

In an HPC project at our organization it is being target to push and pull singularity containers to a gitlab server, using shpc and environment modules to provide simplified user access to software on the cluster. However some of the containers are to be stored on the private registry; while others can be used directly from public registries.

I am interested in other users opinion if this is a good feature to request or contribute. This project seems to meet a number of sought-after features and efforts are appreciated.

So far, in testing, an attempt is made to update the shpc python files in install path for two variables that seem static defined (cannot find config option: https://singularity-hpc.readthedocs.io/en/latest/getting_started/user-guide.html#id3 ):

$ grep -nRi github_url /usr/local/lib/python3.6/site-packages/singularity_hpc-0.0.24-py3.6.egg/shpc
/usr/local/lib/python3.6/site-packages/singularity_hpc-0.0.24-py3.6.egg/shpc/defaults.py:15:# The GitHub repository with recipes
/usr/local/lib/python3.6/site-packages/singularity_hpc-0.0.24-py3.6.egg/shpc/defaults.py:16:github_url = "https://github.com/singularityhub/singularity-hpc"
..
$ grep -A5 -B2 -n 'url =' /usr/local/lib/python3.6/site-packages/singularity_hpc-0.0.24-py3.6.egg/shpc/main/container.py
78-
79-        # Assemble the artifact url
80:        #url = "https://github.com/%s/releases/download/%s/%s.%s.sif" % (
81:        url =  "https://gitlabxyz.nrc-cnrc.gc.ca/singularity-hpc/%s/%s.%s.sif" % (
82-            repo,
83-            github_tag,
84-            prefix,
85-            container_tag,
86-        )

Config options that exist for the base directory paths are local to the host running shpc, for module_base, container_base and namespace options. Thus, setting these doesn't seem to help with our repository access.

[vsoch]

Is it currently supported to use a GitLab repository or container registry on a private network with shpc? Is it possible with a config to update the base repository url?

If you have permission to pull the container outside of shpc (e.g., with singularity natively), it should be possible - so the first thing I would do is produce a singularity command that successfully pulls, show that to me, and then we can figure out the best approach to take.

[nrcfieldsa]

Just as a test we have a CentOS 7 container with miniconda installed and I can pull it as follows:

$ singularity pull docker://registry.azurecr.io/miniconda:latest

A bit further info on what seems to work to some extent: there was no '@' support in podman on this RHEL8 compatible Almalinux test VM where I'm running singularity, docker and shpc commands. It seems that docker-pull(1) commands are aliased to the podman-pull(1) commands, but lack certain syntax. Thus, I've had to use the following settings to get shpc install to recognize my docker private registry.

$ shpc config set gh:gitlabxyz.nrc-cnrc.gc.ca/singularity-hpc/     
$ shpc config set docker:nrc-private-regsitry.azurecr.io

$ diff -u /usr/local/lib/python3.6/site-packages/singularity_hpc-0.0.24-py3.6.egg/shpc{,.bak}/main/modules.py
--- /usr/local/lib/python3.6/site-packages/singularity_hpc-0.0.24-py3.6.egg/shpc/main/modules.py        2022-01-19 19:34:45.675694606 -0500
+++ /usr/local/lib/python3.6/site-packages/singularity_hpc-0.0.24-py3.6.egg/shpc.bak/main/modules.py    2022-01-11 20:45:46.679342526 -0500
@@ -374,7 +374,7 @@

         # We pull by the digest
         if pull_type == "docker":
-            container_uri = "docker://%s:%s" % (config.docker, tag.digest)
+            container_uri = "docker://%s@%s" % (config.docker, tag.digest)
         elif pull_type == "gh":
             container_uri = "gh://%s/%s:%s" % (config.gh, tag.digest, tag.name)

I also had to convert the container from singularity to docker in order to populate as a docker format image. Otherwise, it would complain about the wrong signature/checksum during singularity pull docker://registry even though there was an SIF image upload to the same container name/tag. It just so happens that the registry supports both docker:// and oras:// (OCI) protocols, but you need to store the images separately after conversion.

The container.yaml file used was similar to:

docker: registry.azurecr.io/miniconda
#url: docker://registry.azurecr.io/miniconda:latest
maintainer: '@fieldsa'
description: miniconda
latest:
  0.0.1: "latest"
tags:
  0.0.1: "latest"
filter:
  - "0.0.*"
aliases:
  miniconda: /singularity

It doesn't seem the url: is used and is just a comment, perhaps in a future release it could take this parameter from the file and populate the docker:// host string instead of using the docker: / line. Perhaps it completely ignores the config for docker and gh and just use the string provided in container.yaml instead.

Generally this is encouraging as it's going in the right direction. I can load the module and run miniconda from the Singularity command line in the container.

[vsoch]

So - you shouldn't need to do any more than create a subfolder under registry with the namespace you want, e.g.:

registry/
   registry.azurecr.io/
       miniconda/

and then write a container.yaml in that folder - I think you were close but you need the actual digests for the tags.

docker: registry.azurecr.io/miniconda
url: "https:/...." # some web page for the container
maintainer: '@nrcfieldsa'
description: The container description.
latest:
  latest: sha256:xxxxxxxxxxx
tags:
  latest: sha256:xxxxxxxxxxx
aliases:
  miniconda: /singularity

And make sure to get the correct hashes, and inspect the container inside for the aliases you want to expose. This is a docker pull so I'm not sure why you are trying to edit GitHub / GitLab?

These two settings don't mean anything (there is no docker or gh setting in settings)

$ shpc config set gh:gitlabxyz.nrc-cnrc.gc.ca/singularity-hpc/     
$ shpc config set docker:nrc-private-regsitry.azurecr.io

We could definitely add an oras:// uri for images, but that sounds like a different issue than what we have here.

[vsoch]

It doesn't seem the url: is used and is just a comment

The URL is a user friendly URL for the image. E.g., for a docker URI it would be the page on Docker Hub. It's intended only to be parsed into the docs for a human and is not used to derive the image name. The image name belongs entirely under docker:// given a docker URI.

[vsoch]

Otherwise, it would complain about the wrong signature/checksum during singularity pull docker://registry even though there was an SIF image upload to the same container name/tag. It just so happens that the registry supports both docker:// and oras:// (OCI) protocols, but you need to store the images separately after conversion.

We could easily add an oras:// endpoint / puller if native SIF is your preference.

[nrcfieldsa]

Thanks for clarifying the registry formats and configs.

The reason I was attempting to also set a github_url, was so that if the container.yaml file is pointing to a gh: project - that project could contain the SIF images (git commit && git push), versus using the docker private registry (docker push) or even singularity push where supported by OCI.

We can do either, and as a matter of flexibility I'd like to test both. However, using one alone is enough for most requirements, as we can instruct users which registry to upload to, in order for it to be a simple: module load # on hpc clusters. This will prove handy for in-house code that isn't yet released for public use on GitHub for example, where a container image could be produced an put into standard registries like DockerHub (I think one has been already).

I agree that it would be a great contribution for adding oras:// endpoint - native SIF is a preference since then we could leverage images being produced by researchers on machines w/o root singularity build --fakeroot followed by a singularity push oras://registry

Just to mention: Provided individuals can use the main GitHub repository or setup their own GitLab hooks.. Doing an end-to-end build on the GitHub project with singularity-deploy is a great idea, as then all that is needed is a commit with the latest Dockerfile or Definition file. If I read it correctly the SIF file is produced during the merge request and placed into the /releases on GitHub project, automatically stored with version and release.

[vsoch]

The reason I was attempting to also set a github_url, was so that if the container.yaml file is pointing to a gh: project - that project could contain the SIF images (git commit && git push), versus using the docker private registry (docker push) or even singularity push where supported by OCI.

If you want to use SIF we should add an oras endpoint proper. The original gh URI was created before packages supported SIF, and it would allow pulling a SIF from within an official release using a tool called Singularity Deploy (as you noted above!). What you are trying to do - SIF via oras from GitHub, is not currently supported. But I think we should 100% have it, because certainly I do a lot of deployment there too! The only drawback with using oras / SIF over docker (and pulling to Singularity) is that you cannot pull a SIF with Docker. I think Podman might eventually have support though.

If this sounds good to you I'll add to my TODO to work on - this isn't a work project so likely I'll find some time an evening this week, and latest over the weekend. Let me know if that would be ok!

[nrcfieldsa]

Certainly. I appreciate your efforts and look forward to contributing back what I can, if only in terms of testing / updating ticket with results or further steps.

[vsoch]

Will do ! I'll get started asap - I don't think it should be too much work, just need to make sure to add the proper testing and whatnot. Will ping you when I have something to try!

[vsoch]

okay @nrcfieldsa I have a PR ready for you!

#472

Note that you'll need to install shpc from that branch, and also do pip install -e . again to get a newly released singularity python (spython) as the previous didn't have support for oras. I've also added an example ghcr.io image: https://github.com/singularityhub/github-ci/pkgs/container/github-ci and you should be able to try shpc install ghcr.io/singularityhub/github-ci.

Try it out, tweak the recipe for your use case, and let me know what further steps we need to work on. I'll be back tomorrow evening for another round.

[nrcfieldsa]

Success: test install from branch add/oras using shpc-0.0.39 (PR#472).

## 1. Use shpc-0.0.39 on base OS (AlmaLinux 8.5/Python 3.6.8/Singularity 3.8.5-2.el8/Podman 3.3.1):
######
[nrcfieldsa@container-test singularity-hpc]$ pip3 install -e .   # ok
[nrcfieldsa@container-test singularity-hpc]$ less /home/nrcfieldsa/github.com/singularityhub/singularity-hpc/registry/ghcr.io/singularityhub/github-ci/container.yaml
oras: ghcr.io/singularityhub/github-ci
url: https://github.com/singularityhub/github-ci/pkgs/container/github-ci
maintainer: '@vsoch'
description: An example SIF on GitHub packages to pull with oras
latest:
  latest: sha256:227a917e9ce3a6e1a3727522361865ca92f3147fd202fa1b2e6a7a8220d510b7
tags:
  latest: sha256:227a917e9ce3a6e1a3727522361865ca92f3147fd202fa1b2e6a7a8220d510b7

[nrcfieldsa@container-test singularity-hpc]$ shpc install ghcr.io/singularityhub/github-ci
singularity pull --name /home/nrcfieldsa/github.com/singularityhub/singularity-hpc/modules/ghcr.io/singularityhub/github-ci/la
test/ghcr.io-singularityhub-github-ci-latest-sha256:227a917e9ce3a6e1a3727522361865ca92f3147fd202fa1b2e6a7a8220d510b7.sif oras://
ghcr.io/singularityhub/github-ci@sha256:227a917e9ce3a6e1a3727522361865ca92f3147fd202fa1b2e6a7a8220d510b7
INFO:    Using cached SIF image
INFO:    Using cached SIF image
/home/nrcfieldsa/github.com/singularityhub/singularity-hpc/modules/ghcr.io/singularityhub/github-ci/latest/ghcr.io-singularity
hub-github-ci-latest-sha256:227a917e9ce3a6e1a3727522361865ca92f3147fd202fa1b2e6a7a8220d510b7.sif
Module ghcr.io/singularityhub/github-ci:latest was created.

[nrcfieldsa@container-test singularity-hpc]$ export MODULEPATH=$(pwd)/modules
[nrcfieldsa@container-test singularity-hpc]$ module avail

----------------------------- /home/nrcfieldsa/github.com/singularityhub/singularity-hpc/modules -----------------------------
   ghcr.io/singularityhub/github-ci/latest/module

[nrcfieldsa@container-test singularity-hpc]$ module load ghcr.io/singularityhub/github-ci
[nrcfieldsa@container-test singularity-hpc]$ github-ci-run
Hold me closer... tiny container :) :D

[nrcfieldsa@container-test singularity-hpc]$ github-ci-shell
Singularity>
Singularity> ls -l /singularity
lrwxrwxrwx    1 root     root            24 Oct 23 18:59 /singularity -> .singularity.d/runscript
Singularity> cat /singularity
#!/bin/sh

echo "Hold me closer... tiny container :) :D"

Singularity> exit


## 2. Use the included ./Dockerfile to test running shpc-0.0.39 inside a singularity container
#####

# build a container with lmod for singularity-hpc
[nrcfieldsa@container-test singularity-hpc]$ docker build -t shpc-oras-docker.tar .
[nrcfieldsa@container-test singularity-hpc]$ docker save -o shpc-oras.sif localhost/shpc-oras-docker.tar .

[nrcfieldsa@container-test singularity-hpc]$ singularity build -s shpc-oras shpc-oras.sif
INFO:    Starting build...
INFO:    Verifying bootstrap image shpc-oras.sif
WARNING: integrity: signature not found for object group 1
WARNING: Bootstrap image could not be verified, but build will continue.
INFO:    Creating sandbox directory...
INFO:    Build complete: shpc-oras

[nrcfieldsa@container-test singularity-hpc]$ singularity shell -w shpc-oras/
Singularity> shpc --version
0.0.39
Singularity> ls
CHANGELOG.md  Dockerfile      LICENSE      README.md  entrypoint.sh  registry  shpc       shpc-oras-docker.tar
CITATION.cff  Dockerfile.tcl  MANIFEST.in  docs       paper          setup.py  shpc-oras  shpc-oras.sif
Singularity> touch /code/test # writable? ok
Singularity> rm /code/test

#####################################################################################################
Singularity> shpc install ghcr.io/singularityhub/github-ci
singularity pull --name /code/modules/ghcr.io/singularityhub/github-ci/latest/ghcr.io-singularityhub-github-ci-latest-sha256:227
a917e9ce3a6e1a3727522361865ca92f3147fd202fa1b2e6a7a8220d510b7.sif oras://ghcr.io/singularityhub/github-ci@sha256:227a917e9ce3a6e
1a3727522361865ca92f3147fd202fa1b2e6a7a8220d510b7
INFO:    Downloading oras image
INFO:    Downloading oras image
/code/modules/ghcr.io/singularityhub/github-ci/latest/ghcr.io-singularityhub-github-ci-latest-sha256:227a917e9ce3a6e1a3727522361
865ca92f3147fd202fa1b2e6a7a8220d510b7.sif
Module ghcr.io/singularityhub/github-ci:latest was created.
Singularity> shpc --version
0.0.39
Singularity> shpc show ghcr.io/singularityhub/github-ci
oras: ghcr.io/singularityhub/github-ci
url: https://github.com/singularityhub/github-ci/pkgs/container/github-ci
maintainer: '@vsoch'
description: An example SIF on GitHub packages to pull with oras
latest:
  latest: sha256:227a917e9ce3a6e1a3727522361865ca92f3147fd202fa1b2e6a7a8220d510b7
tags:
  latest: sha256:227a917e9ce3a6e1a3727522361865ca92f3147fd202fa1b2e6a7a8220d510b7
######################################################################################################

Singularity> module load ghcr.io/singularityhub/github-ci
bash: module: command not found
Singularity> . /etc/profile.d/modules.sh
Singularity> export MODULEPATH=/code/modules:$MODULEPATH
Singularity> module avail

-------------------------------------------------------- /code/modules ---------------------------------------------------------
   ghcr.io/singularityhub/github-ci/latest/module

Singularity> module load ghcr.io/singularityhub/github-ci
[nrcfieldsa@container-test singularity-hpc]$ set|less
..
github-ci-inspect-deffile ()
{
    singularity ${SINGULARITY_OPTS} inspect ${SINGULARITY_COMMAND_OPTS} -d /code/modules/ghcr.io/singularityhub/github-ci/latest/ghcr.io-singularityhub-github-ci-latest-sha256:227a917e9ce3a6e1a3727522361865ca92f3147fd202fa1b2e6a7a8220d510b7.sif
}
github-ci-inspect-runscript ()
{
    singularity ${SINGULARITY_OPTS} inspect ${SINGULARITY_COMMAND_OPTS} -r /code/modules/ghcr.io/singularityhub/github-ci/latest/ghcr.io-singularityhub-github-ci-latest-sha256:227a917e9ce3a6e1a3727522361865ca92f3147fd202fa1b2e6a7a8220d510b7.sif
}
github-ci-run ()
{
    singularity ${SINGULARITY_OPTS} run ${SINGULARITY_COMMAND_OPTS} -B /code/modules/ghcr.io/singularityhub/github-ci/latest/99-shpc.sh:/.singularity.d/env/99-shpc.sh /code/modules/ghcr.io/singularityhub/github-ci/latest/ghcr.io-singularityhub-github-ci-latest-sha256:227a917e9ce3a6e1a3727522361865ca92f3147fd202fa1b2e6a7a8220d510b7.sif $@
}
github-ci-shell ()
{
    singularity ${SINGULARITY_OPTS} shell ${SINGULARITY_COMMAND_OPTS} -s /bin/sh -B /code/modules/ghcr.io/singularityhub/github-ci/latest/99-shpc.sh:/.singularity.d/env/99-shpc.sh /code/modules/ghcr.io/singularityhub/github-ci/latest/ghcr.io-singularityhub-github-ci-latest-sha256:227a917e9ce3a6e1a3727522361865ca92f3147fd202fa1b2e6a7a8220d510b7.sif
}

For next test: I will run shpc config again for my local registry and try with a private container registry using OCI.

[vsoch]

Sounds good!

For next test: I will run shpc config again for my local registry and try with a private container registry using OCI.

You shouldn't need to use shpc config for anything - you just need to create the corresponding container.yaml and folder that has the registry URI there.

[nrcfieldsa]

The issue I get in my test with private OCI container registry is as follows:

[nrcfieldsa@container-test singularity-hpc]$ cat registry/lolnrc/container.yaml
oras: registry.azurecr.io/lolnrc
url: oras://registry.azurecr.io/lolnrc:
maintainer: '@nrcfieldsa'
description: Cowsay NRC
latest:
  latest: "sha256:d652da62c85a2d415c8e2d44a2547e14aa66aa8a0107881f83e255dce7c083c9"
tags:
  latest: "sha256:d652da62c85a2d415c8e2d44a2547e14aa66aa8a0107881f83e255dce7c083c9"
aliases:
  lolnrc: /singularity

[nrcfieldsa@container-test singularity-hpc]$ shpc install lolnrc
singularity pull --name /home/nrcfieldsa/github.com/singularityhub/singularity-hpc/modules/registry.azurecr.io/lolnrc/latest/registry.azurecr.io-lolnrc-latest-sha256:d652da62c85a2d415c8e2d44a2547e14aa66aa8a0107881f83e255dce7c083c9.sif oras://registry.azurecr.io/lolnrc@sha256:d652da62c85a2d415c8e2d44a2547e14aa66aa8a0107881f83e255dce7c083c9
FATAL:   While pulling image from oci registry: error fetching image to cache: failed to get checksum for oras://registry.azurecr.io/lolnrc@sha256:d652da62c85a2d415c8e2d44a2547e14aa66aa8a0107881f83e255dce7c083c9: while resolving reference: pulling from host registry.azurecr.io failed with status code [manifests sha256:d652da62c85a2d415c8e2d44a2547e14aa66aa8a0107881f83e255dce7c083c9]: 500 Internal Server Error
FATAL:   While pulling image from oci registry: error fetching image to cache: failed to get checksum for oras://registry.azurecr.io/lolnrc@sha256:d652da62c85a2d415c8e2d44a2547e14aa66aa8a0107881f83e255dce7c083c9: while resolving reference: pulling from host registry.azurecr.io failed with status code [manifests sha256:d652da62c85a2d415c8e2d44a2547e14aa66aa8a0107881f83e255dce7c083c9]: 500 Internal Server Error
There was an issue pulling None


[nrcfieldsa@container-test singularity-hpc]$ cat registry/lolnrc/container.yaml
oras: registry.azurecr.io/lolnrc
url: oras://registry.azurecr.io/lolnrc:latest
maintainer: '@nrcfieldsa'
description: Cowsay NRC
latest:
  0.0.1: latest
tags:
  0.0.1: latest
aliases:
  lolnrc: /singularity

[nrcfieldsa@container-test singularity-hpc]$ shpc install lolnrc
singularity pull --name /home/nrcfieldsa/github.com/singularityhub/singularity-hpc/modules/registry.azurecr.io/lolnrc/0.0.1/registry.azurecr.io-lolnrc-0.0.1-latest.sif oras://registry.azurecr.io/lolnrc@latest
FATAL:   While pulling image from oci registry: error fetching image to cache: failed to get checksum for oras://registry.azurecr.io/lolnrc@latest: while resolving reference: invalid checksum digest format
FATAL:   While pulling image from oci registry: error fetching image to cache: failed to get checksum for oras://registry.azurecr.io/lolnrc@latest: while resolving reference: invalid checksum digest format
There was an issue pulling None

# when I pushed the container as a test:
$ sha256sum lolnrc.sif
d652da62c85a2d415c8e2d44a2547e14aa66aa8a0107881f83e255dce7c083c9 lolnrc.sif
$ singularity push lolnrc.sif oras://registry.azurecr.io/lolnrc:latest
INFO:    Upload complete
$ singularity push lolnrc.sif oras://registry.azurecr.io/lolnrc@sha256:d652da62c85a2d415c8e2d44a2547e14aa66aa8a0
107881f83e255dce7c083c9
INFO:    Upload complete

I don't think my version of singularity likes the @latest but needs :latest instead to d/l ok with-out running into a checksum error. It doesn't seem to pull OK with the sha256 schema even if singularity is called directly and attempts to parse the checksum value when specifying @sha256:...

# pull with "latest":
[nrcfieldsa@container-test registry]$ singularity pull --disable-cache --name /home/nrcfieldsa/github.com/singularityhub/singularity-hpc/modules/registry.azurecr.io/lolnrc/0.0.1/registry.azurecr.io-lolnrc-0.0.1-latest.sif oras://registry.azurecr.io/lolnrc@latest
FATAL:   While pulling image from oci registry: error fetching image to cache: failed to get checksum for oras://registry.azurecr.io/lolnrc@latest: while resolving reference: invalid checksum digest format
[nrcfieldsa@container-test registry]$ singularity pull --disable-cache --name /home/nrcfieldsa/github.com/singularityhub/singularity-hpc/modules/registry.azurecr.io/lolnrc/0.0.1/registry.azurecr.io-lolnrc-0.0.1-latest.sif oras://registry.azurecr.io/lolnrc:latest
INFO:    Downloading oras image

# pull with sha256:
[nrcfieldsa@container-test singularity-hpc]$ singularity pull --disable-cache --name /home/nrcfieldsa/github.com/singularityhub/singularity-hpc/modules/registry.azurecr.io/lolnrc/0.0.1/registry.azurecr.io-lolnrc-0.0.1-latest.sif oras://registry.azurecr.io/lolnrc:sha256:d652da62c85a2d415c8e2d44a2547e14aa66aa8a0107881f83e255dce7c083c9
FATAL:   While pulling image from oci registry: error fetching image to cache: failed to get checksum for oras://registry.azurecr.io/lolnrc:sha256:d652da62c85a2d415c8e2d44a2547e14aa66aa8a0107881f83e255dce7c083c9: while resolving reference: pulling from host registry.azurecr.io failed with status code [manifests sha256:d652da62c85a2d415c8e2d44a2547e14aa66aa8a0107881f83e255dce7c083c9]: 500 Internal Server Error
[nrcfieldsa@container-test singularity-hpc]$ singularity pull --disable-cache --name /home/nrcfieldsa/github.com/singularityhub/singularity-hpc/modules/registry.azurecr.io/lolnrc/0.0.1/registry.azurecr.io-lolnrc-0.0.1-latest.sif oras://registry.azurecr.io/lolnrc@sha256:d652da62c85a2d415c8e2d44a2547e14aa66aa8a0107881f83e255dce7c083c9
FATAL:   While pulling image from oci registry: error fetching image to cache: failed to get checksum for oras://registry.azurecr.io/lolnrc@sha256:d652da62c85a2d415c8e2d44a2547e14aa66aa8a0107881f83e255dce7c083c9: while resolving reference: pulling from host registry.azurecr.io failed with status code [manifests sha256:d652da62c85a2d415c8e2d44a2547e14aa66aa8a0107881f83e255dce7c083c9]: 500 Internal Server Error

Arguably GitHub may make a more friendly backend - however, is this something to do with OCI artifacts standard when specifying '@', and not specific to the registry in question or singularity CLI?

[nrcfieldsa]

You shouldn't need to use shpc config for anything - you just need to create the corresponding container.yaml and folder that has the registry URI there.

Note my test was done with-out changing any config values and just adding the container.yaml into the checked-out project ./registry dir.

[vsoch]

Can you show me the pull command that works?

I've actually run into that issue too with tags - if you don't provide a tag it messes things up a bit.