How to build a singularity image from a local docker image (not a registry image)?

[bbarker]

Version of Singularity:

2.5.1

Expected behavior

Documented way to create a singularity image from a local docker image (not an image on a registry)

Looked at documentation here.

Actual behavior

brandon@beb82dell0:~/Downloads$ docker images | grep testing8
nix_ubuntu                       testing8                         bea3190832c4        25 hours ago        10.1GB
brandon@beb82dell0:~/Downloads$ singularity build nix_ubuntu_testing8.img docker://nix_ubuntu:testing8
Docker image path: index.docker.io/library/nix_ubuntu:testing8
ERROR UNAUTHORIZED: authentication required
ERROR Check existence, naming, and permissions
Cleaning up...

Steps to reproduce behavior

See above

[GodloveD]

Hi @bbarker. I'm going to copy and paste the description from @dctrud here.

# Start a docker registry
$ docker run -d -p 5000:5000 --restart=always --name registry registry:2

# Push local docker container to it
$ docker tag alpine localhost:5000/alpine
$ docker push localhost:5000/alpine

# Create def file for singularity like this..
# (add your modifications)
Bootstrap: docker
Registry: http://localhost:5000
Namespace:
From: alpine:latest

# Build singularity container
$ sudo SINGULARITY_NOHTTPS=1 singularity build alpine.simg def

You're right, this should really be in the Singularity docs. Do you feel like submitting a PR to https://github.com/singularityware/singularityware.github.io/tree/docs/2.5? If not, I can try to get to it when I have a minute. Thanks!

[holgerbrandl]

@GodloveD @psteinb It seems that this approach requires root access on the host. Without being root it (2.4-dist) just says:

> SINGULARITY_NOHTTPS=1 singularity build foo.simg def
Using container recipe deffile: def
ERROR: You must be the root user to build from a definition file

Is there another workaround for this step which does not require sudo?

[GodloveD]

I'm going to ping @dctrud and @vsoch on this. I know you can do something like this without sudo:

$ singularity build my_image docker://me/my_image

But don't know if you can supply a custom URI (localhost) with that syntax.

[dtrudg]

It should work like this...

SINGULARITY_NOHTTPS=1 singularity build alpine.simg docker://localhost:5000/alpine:latest

[holgerbrandl]

Thanks @dctrud for the advice. Works great for me.

[bilke]

Just a reference to #2255 that this does not work with 3.0 at the moment.

[jscook2345]

Closing since this is now #2255

[dahlo]

Just a note to anyone else trying to do this, you can get this to work by specifying docker-daemon:// instead of just docker:// in your singularity build command. No need to start your own docker registry and pushing it there. As long as you have the docker image on your computer it will use the cached layers in your docker daemon directly.

Using OPs command:
singularity build nix_ubuntu_testing8.img docker-daemon://nix_ubuntu:testing8

It would be nice if it could be mentioned in the Overview list of targets in https://sylabs.io/guides/3.5/user-guide/build_a_container.html

[dtrudg]

The original question here was on a 2.x version of Singularity - for the benefit of anyone using 2.6 still, docker-daemon is not available there. It is a 3.x feature.

@dahlo - yes we could add docker-daemon to the page you link. There are other (less commonly usd) sources not mentioned in that quick list. docker-archive and oci archives are supported too.

[efratushava]

@GodloveD and everyone,
I also encountered this problem.

I'm using singularity 2.6.0-dist, without root permissions (I don't have docker installed).

How can I run a local docker image?

Running: singularity run docker://efrat/Dockerfile (My dockerfile is in the efrat directory)
Gives out: ERROR : Authentication error, exiting.

What can I do?

[dtrudg]

@efratushava - In 2.x you have to build the docker image with docker somewhere, push it to a registry (either DockerHub, something else, or a local registry), than have Singularity pull from that registry, per the comment above. It is not possible to just build or run from Dockerfile

If you need to keep things local, you will need to run a local docker registry somewhere you do have docker, per the comment above:

sylabs/singularity#1537 (comment)

[durai23]

Just a note to anyone else trying to do this, you can get this to work by specifying docker-daemon:// instead of just docker:// in your singularity build command. No need to start your own docker registry and pushing it there. As long as you have the docker image on your computer it will use the cached layers in your docker daemon directly.

Using OPs command:
singularity build nix_ubuntu_testing8.img docker-daemon://nix_ubuntu:testing8

It would be nice if it could be mentioned in the Overview list of targets in https://sylabs.io/guides/3.5/user-guide/build_a_container.html

@dahlo while this works, I have to be root for it to work. If I run as regular user I get:

INFO: Starting build... FATAL: While performing build: conveyor failed to get: Error loading image from docker engine: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.22/images/get?names=alpine%3Alatest": dial unix /var/run/docker.sock: connect: permission denied

This is singularity version 3.5.2

[dahlo]

@durai23 It looks like your user is not a member of the docker group. If you add it as a member you should be able to do it without sudo.

Cheers

[5MI7th3MI6]

Hello,

I was able to successfully take the Docker Image and convert it to singularity .sif image. Using this command SINGULARITY_NOHTTPS=1 singularity build deephyperx.sif docker://localhost:5000/deephyperx:latest However, I get this error /bin/sh: 0: Can't open start.sh. Any ideas on how I can fix it?

Also, when I run this container on docker with this command docker run -p 9999:8097 -ti --rm -v pwd:/workspace/DeepHyperX/ deephyperx:latest it works without any errors.

https://github.com/nshaud/DeepHyperX

[vsoch]

Singularity does not have any concept of WORKDIR . So either you need to update the entrypoint / cmd to target the exact path to start.sh, or you need to do a singularity exec to the container to hit it. It should be something like

$ singularity exec deephyperx.sif /workspace/DeepHyperX/start.sh 

Also, if you need ports you probably want to check out instances, and you'd need to take a similar approach - a %startscript won't be populated by a Docker conversion, so you'd need to start it and then exec that command to start the service. You can also check out singularity-compose if you want a simpler interaction.

[5MI7th3MI6]

singularity exec deephyperx.sif /workspace/DeepHyperX/start.sh 

Thank you for the response. Gave your command a try and it says permission denied.

[vsoch]

I would guess you are trying to do something with a port but you haven't created an instance that has permission to. You basically want to duplicate that exec command, but with a singularity instance.

[5MI7th3MI6]

I would guess you are trying to do something with a port but you haven't created an instance that has permission to. You basically want to duplicate that exec command, but with a singularity instance.

Hello @vsoch ,
If I understand it correctly, does that mean I need to configure the def?

[vsoch]

Yes, you likely want to edit that startscript to have the start command, unless you are okay using exec. If you are having trouble I can probably help you test it out over the weekend with singularity-compose: https://singularityhub.github.io/singularity-compose/#/. which will just provide an easy wrapper to doing all this. But you'd have to be willing to share your container recipe to some extent. if not, check out some of the examples https://github.com/singularityhub/singularity-compose-examples.

[5MI7th3MI6]

y want to edit that startscript to have the start comma

@vsoch No problem, I can meet with you over the weekend. In the meantime, I will take a look at the singularity-compose. Thank you.

[5MI7th3MI6]

Hello, @vsoch

[5MI7th3MI6]

Hello @vsoch

Is it possible we can talk or meet? When you are available? Thank you in advance.

[vsoch]

If you’d like to share your recipe I can reproduce I can offer to help when I have some time, but I’m not able to offer time to meet or talk with you personally.

[5MI7th3MI6]

If you’d like to share your recipe I can reproduce I can offer to help when I have some time, but I’m not able to offer time to meet or talk with you personally.

Do you mean the docker container?

[vsoch]

Yes or the recipe (Dockerfile). I just need to be able to reproduce your error.

[5MI7th3MI6]

Yes or the recipe (Dockerfile). I just need to be able to reproduce your error.

I used this Docker container to build it locally.
https://github.com/nshaud/DeepHyperX

[vsoch]

here you go, see the README for instructions. https://github.com/singularityhub/singularity-compose-examples/tree/master/v2.0/deephyperx. If you use --debug you can see the command that is generated for the instance, in case you don't want to use singularity-compose (or otherwise want to modify it). Singularity compose docs are here if you need singularityhub.github.io/singularity-compose/. Enjoy!

[5MI7th3MI6]

@vsoch

Do you know if it is possible to do it without singularity-compose?

[vsoch]

sure, just turn on debug for singularity compose and copy paste all the long hairy commands, run those!

[5MI7th3MI6]

sure, just turn on debug for singularity compose and copy paste all the long hairy commands, run those!

Sorry, what commands do you mean. The reason we are not able to use singularity-compose at the moment is because our HPC does not have sudo privileges enabled and for me to get singularity-compose on it I need sudo privileges.

[vsoch]

Then use your local machine?

[5MI7th3MI6]

Then use your local machine?

Only if I could.

Our research is based on HPC and I have emailed them about it but I am not sure if they will give us sudo privileges or install it for us.

[5MI7th3MI6]

@vsoch

Is it possible we can meet? I can show you the error I am having. Or I can explain it here if it okay with you?

[vsoch]

I would suggest first going to the hpcng or Sylabs Slack and asking for help there - there are developers there paid to assist with this kind of thing. I've already taken a few hours to put together that example and tell you how to get the commands you seek, and I've just about reached my threshold for providing this kind of help. But I assure you that the Singularity developers can assist you further.

[5MI7th3MI6]

I would suggest first going to the hpcng or Sylabs Slack and asking for help there - there are developers there paid to assist with this kind of thing. I've already taken a few hours to put together that example and tell you how to get the commands you seek, and I've just about reached my threshold for providing this kind of help. But I assure you that the Singularity developers can assist you further.

@vsoch No problem, thank you so much for your help!

[Zoidmania]

Checking in here, following up on @dtrudg's comment, you don't need to stand up a local Docker registry. If you have the tarball of an image (a la docker save image:tag -o file.tar), you can simply build an image file from that:

apptainer build new-image.sif docker-archive:file.tar
apptainer run new-image.sif

Tested with apptainer version 1.1.7-1.el8.

[FerranAgulloLopez]

It is documented here -> https://docs.sylabs.io/guides/3.0/user-guide/singularity_and_docker.html#singularity-and-docker

Specifically in sections:

• Locally Available Images: Cached by Docker
• Mandatory Header Keywords: Locally Boostrapped