Remote container with Singularity

[statiksof]

Is it possible to use singularity containers instead of Docker? We only provide Singularity on our remote machines.

[Chuxel]

Not today. Only Docker is fully supported today. Kubernetes has attach support and Podman works with some limitations.

[statiksof]

@Chuxel thanks for the quick answer. Actually, I though that I can overcome the glibc issue by using containers. Is there any other workaround to make the remote ssh works? I have glibc 2.12.

[Chuxel]

Not today - Node, V8, and even Chrome require recent versions of glibc unfortunately.

[statiksof]

Just an update, I can start the vscode-server inside the singularity container. It starts without any issue. The problem now is how to forward ssh or config the ssh extension locally to use this server.

[oschulz]

I would love to see support for Singularity containers, together with other container systems like Shifter and Charliecloud that are popular in scientific (resp. HPC/HTC) computing environments.

I currently use a very clunky workaround:

• I use a custom SSH script (via VSCode config option"remote.SSH.path") that sets an env var LC__SINGULARITY_IMAGE (different images can be chosen depending on the name of the target system) and then does exec ssh "$@". Env vars starting with LC_* (and only those) are usually forwarded by SSH.

• I modify "$HOME/.vscode-server/bin/*/server.sh", and add change "$ROOT/node ..." to singularity run "$LC__SINGULARITY_IMAGE" $ROOT/node ..."

It works, but it's not very convenient. "server.sh" needs to be modified after every update of vscode-server, and since the server instance gets reused, only one container instance can be active on a given remote system at the same time.

[oschulz]

A first step to make it easier for users to use custom container runtimes might be to add a configuration option like "remote.SSH.runwith". It would allow the user to specify a wrapper script for the vscode server. That user-specified wrapper script would be required to end with exec $@, so that "remote.SSH.runwith" = "run_in_my_container.sh" would result in run_in_my_container.sh "$ROOT/node" ${INSPECT:-} "$ROOT/out/vs/server/main.js" "$@" or run_in_my_container.sh /path/to/server.sh ....

If VS Code would also set some env vars before calling that wrapper script, forwarding information like workspace path, then the wrapper script could decide which container image to run, and how, depending on the application.

[statiksof]

@oschulz thank you for sharing your workaround. I finally got it to work after multiple refinements. For me, forwarding the LC* didn't work. So, I spent most of the time modifying server.sh. I used a default centos 7 image just for testing, there are some warnings when the agent starts, I am thinking to provide dedicated images for that purpose.

[oschulz]

For me, forwarding the LC* didn't work.

Unfortunately, that depends on the SSH server config - I guess some sites disable it, and users don't have control over the server config, of course.

[RobertRosca]

The instructions for using Podman with the remote containers are here: https://code.visualstudio.com/docs/remote/containers#_can-i-use-podman-instead-of-docker

It just suggests pointing the docker executable path to the podman path, and it works as podman and docker have similar CLI commands.

One solution would be to create a wrapper for Singularity which parses and 'translates' the docker commands into Singularity commands. This is similar to the workaround mentioned here int this comment above #3066 (comment) but it's slightly less clunky.

I'll add this to my list of cool things to maybe work on, but probably won't get to it. If anybody here is interested I'm willing to help out though.

[oschulz]

Thanks @RobertRosca , I'll definitely try that, such a wrapper should be possible. Will post if it works.

[david-macleod]

Another workaround (#4474) using a modified version of the remote extension (which will hopefully make it into the default extension soon) allows specifying the singularity exec in .ssh/config e.g.

Host blah
    User blah
    HostName blah
    RemoteCommand "/usr/local/bin/singularity exec $CONTAINER_IMAGE /bin/bash"

[oschulz]

I guess the ideal solution would be a config option for remote-ssh that allows the user to specify an arbitrary command that gets inserted before "$ROOT/node [...]. That way, one could use workspace-dependent containers, etc.

Was trying to prototype something like that, but to my knowledge the source code of the remote-ssh extension isn't available in unmangled form. Should be trivial to do for the VS code team, though.