Singularity runtime

[tsibley]

Add a new runner for a Singularity runtime based on the Docker runtime

It's been my intention to support Singularity since day 1, but I never got around to implementing it because usage seemed little to non-existent amongst Nextstrain users I knew about. (Though maybe that's a chicken-or-the-egg problem!) We've since had users report their ad-hoc use of Singularity with Nextstrain, however, and Singularity is often the only containerization method supported by HPC systems. So it seems important to support.

Resolves #2.

Testing

You can test this yourself using a source checkout of this PR, or with the standalone build produced by this PR:

curl -fsSL --proto '=https' https://nextstrain.org/cli/installer/linux \
    | DESTINATION=/tmp/cli-test bash -s pr-build/248

(Download will be much slower than a normal release, but that's ok for dev.)

No need to run the two final commands the installer suggests at the end.

Then get started testing it with:

/tmp/cli-test/nextstrain setup singularity

Checks

• check-setup works
• setup works
• update works
• build works
• view works
• shell works
• Tests pass locally
• Checks pass

[j23414]

Nice to see, especially for HPC users where Docker is not allowed for security reasons.

Although I'll be curious how you handle sysadmin defined bind paths.

[tsibley]

Although I'll be curious how you handle sysadmin defined bind paths.

The runner uses Singularity's --contain option to avoid most system-defined bind paths. Per the default singularity.conf's commentary on the system bind path config:

# NOTE: these are ignored if singularity is invoked with --contain except
# for /etc/hosts and /etc/localtime. When invoked with --contain and --net,
# /etc/hosts would contain a default generated content for localhost resolution.

So I think system-defined bind paths should be a non-issue!

[tsibley]

Nice to see, especially for HPC users where Docker is not allowed for security reasons.

Yep! The Conda runtime is another good option for HPC, I think, though not as isolated as the containerization provided by Singularity.

[j23414]

The Conda runtime is another good option for HPC, I think, though not as isolated as the containerization provided by Singularity.

Nod, or this this workaround

[tsibley]

To test this, the easiest way is to download the standalone archive built by CI and available as a CI artifact.

Artifacts can only be downloaded by authenticated users per GitHub's policy, so you'll either need to download the archive in your browser:

https://github.com/nextstrain/cli/suites/10229277255/artifacts/501436118

or have credentials in your ~/.netrc for an authenticated curl invocation:

curl -fsSL --netrc https://api.github.com/repos/nextstrain/cli/actions/artifacts/501436118/zip > standalone-x86_64-unknown-linux-gnu.zip

Once you have standalone-x86_64-unknown-linux-gnu.zip, then unpack it and setup the runtime:

mkdir -p cli-test
funzip < standalone-x86_64-unknown-linux-gnu.zip | tar xzvp -C cli-test
./cli-test/nextstrain setup singularity

[tsibley]

Nod, or this this workaround

Sure, though this runtime would obsolete that workaround…

[tsibley]

New, simpler way to install the standalone build for this PR:

curl -fsSL --proto '=https' https://nextstrain.org/cli/installer/linux \
    | DESTINATION=/tmp/cli-test bash -s pr-build/248

(Download will be much slower than a normal release, but that's ok for dev.)

No need to run the two final commands the installer suggests at the end.

Then get started testing it with:

/tmp/cli-test/nextstrain setup singularity

[tsibley]

Rebased onto latest master branch to account for #250.

[tsibley]

Per Slack, I'm going to merge this today unless there are objections. It's worked in my testing, incl. on the Hutch's cluster. And I think @j23414 reported it worked for her too. I'm sure there will be things to fix that come up as we use it, so good to get it out there and available. I don't intend to add it yet to our main installation docs, but it will be noted in the Nextstrain CLI docs, and we can offer it in situations it might be useful.