Add a new runner for a Singularity runtime based on the Docker runtime

It's been my intention to support Singularity since day 1, but I never
got around to implementing it because usage seemed little to
non-existent amongst Nextstrain users I knew about.  (Though maybe
that's a chicken-or-the-egg problem!)  We've since had users report
their ad-hoc use of Singularity with Nextstrain, however, and
Singularity is often the only containerization method supported by HPC
systems.  So it seems important to support.

Resolves <#2>.

doc/aws-batch.md

@@ -19,13 +19,13 @@ Batch job, monitors the job status, streams the job logs to your terminal, and
 downloads build results back to the `zika-tutorial/` directory.
 
 The interface aims to be very similar to that of local builds (run in the
-Docker, Conda, or ambient runtimes), so the `nextstrain build` command stays in
-the foreground and result files are written back directly to the local build
-directory.  Alternatively, you can specify the `--detach` option to run AWS
-Batch builds in the background once they're submitted.  The Nextstrain CLI will
-tell you how to reattach to the build later to view the logs and download the
-results.  If you forget to use the `--detach` option, you can press Control-Z
-to detach at any point once the build is submitted.
+Docker, Conda, Singularity, or ambient runtimes), so the `nextstrain build`
+command stays in the foreground and result files are written back directly to
+the local build directory.  Alternatively, you can specify the `--detach`
+option to run AWS Batch builds in the background once they're submitted.  The
+Nextstrain CLI will tell you how to reattach to the build later to view the
+logs and download the results.  If you forget to use the `--detach` option, you
+can press Control-Z to detach at any point once the build is submitted.
 
 [AWS Batch]: https://aws.amazon.com/batch/
 [`zika-tutorial/` directory]: https://github.com/nextstrain/zika-tutorial

doc/installation.md

@@ -57,21 +57,22 @@ example above.
 
 The Nextstrain CLI provides a consistent interface for running and visualizing
 Nextstrain pathogen builds across several different computing environments,
-such as [Docker][], [Conda][], and [AWS Batch][].  Each computing environment
-provides specific versions of Nextstrain's software components and is
-responsible for running Nextstrain's programs like [Augur][] and [Auspice][].
-For this reason, the different computing environments are called "runners" by
-the CLI.
+such as [Docker][], [Conda][], [Singularity](#singularity), and [AWS Batch][].
+Each computing environment provides specific versions of Nextstrain's software
+components and is responsible for running Nextstrain's programs like [Augur][]
+and [Auspice][].  For this reason, the different computing environments are
+called "runners" by the CLI.
 
 At least one of these computing environments, or runners, must be setup in
 order for many of `nextstrain`'s subcommands to work, such as `nextstrain
 build` and `nextstrain view`.
 
 The default runner is Docker, using the [nextstrain/base][] container image.
 Containers provide a tremendous amount of benefit for scientific workflows by
-isolating dependencies and increasing reproducibility.  However, they're not
-always appropriate, so a Conda runner and "ambient" runner are also supported.
-The installation and setup of supported runners is described below.
+isolating dependencies and increasing reproducibility.  However, Docker is not
+always available or appropriate, so a Conda runner, Singularity runner, and
+"ambient" runner are also supported.  The installation and setup of supported
+runners is described below.
 
 [nextstrain/base]: https://github.com/nextstrain/docker-base
 
@@ -110,6 +111,31 @@ On macOS and Linux, run `nextstrain setup conda` to get started.
 This runner is not directly supported on Windows, but you can use [WSL2][] to
 "switch" to Linux and run the above setup command.
 
+### Singularity
+
+Singularity is a container system freely-available for Linux platforms.  It is
+commonly available on institutional HPC systems as an alternative to Docker,
+which is often not supported on such systems.  When you use Singularity with
+the Nextstrain CLI, you don't need to install any other Nextstrain software
+dependencies as validated versions are already bundled into a container image
+by the Nextstrain team.
+
+Run `nextstrain setup singularity` to get started.
+
+Note that the Singularity project forked into two separate projects in late
+2021: [SingularityCE][] under [Sylabs][] and [Apptainer][] under the [Linux
+Foundation][].  Either fork should work with Nextstrain CLI, as both projects
+still provide very similar interfaces and functionality via the `singularity`
+command.  You can read [Sylab's announcement][] and [Apptainer's
+announcement][] for more information on the fork.
+
+[SingularityCE]: https://sylabs.io/singularity/
+[Sylabs]: https://sylabs.io/
+[Apptainer]: https://apptainer.org
+[Linux Foundation]: https://www.linuxfoundation.org/
+[Sylab's announcement]: https://sylabs.io/2022/06/singularityce-is-singularity/
+[Apptainer's announcement]: https://apptainer.org/news/community-announcement-20211130
+
 ### Ambient
 
 The "ambient" runner allows you to use the Nextstrain CLI with your own ambient
@@ -165,6 +191,10 @@ based on what's available.  You should see output similar to the following:
     ✔ yes: augur is installed and runnable
     ✔ yes: auspice is installed and runnable
 
+    # singularity is supported
+    ✔ yes: singularity is installed
+    ✔ yes: singularity works
+
     # ambient is not supported
     ✔ yes: snakemake is installed and runnable
     ✘ no: augur is installed and runnable
@@ -175,18 +205,18 @@ based on what's available.  You should see output similar to the following:
     ✘ no: job queue "nextstrain-job-queue" exists
     ✘ no: S3 bucket "nextstrain-jobs" exists
 
-    All good!  Supported Nextstrain environments: docker, conda
+    All good!  Supported Nextstrain environments: docker, conda, singularity
 
     Setting default environment to docker.
 
 If the output doesn't say "All good!" and list at least one supported
-Nextstrain computing environment (typically Docker, Conda, or ambient), then
-something may be wrong with your installation.
+Nextstrain computing environment (typically Docker, Conda, Singularity, or
+ambient), then something may be wrong with your installation.
 
 The default is written to the _~/.nextstrain/config_ file.  If multiple
 environments are supported, you can override the default for specific runs
-using command-line options such as `--docker`, `--conda`, `--ambient`, and
-`--aws-batch`, e.g. `nextstrain build --ambient …`.
+using command-line options such as `--docker`, `--conda`, `--singularity`,
+`--ambient`, and `--aws-batch`, e.g. `nextstrain build --ambient …`.
 
 
 [Augur]: https://github.com/nextstrain/augur

nextstrain/cli/argparse.py

@@ -178,12 +178,12 @@ def runner_module_argument(name: str) -> RunnerModule:
     >>> runner_module_argument("invalid")
     Traceback (most recent call last):
         ...
-    argparse.ArgumentTypeError: invalid runtime name: 'invalid'; valid names are: 'docker', 'conda', 'ambient', 'aws-batch'
+    argparse.ArgumentTypeError: invalid runtime name: 'invalid'; valid names are: 'docker', 'conda', 'singularity', 'ambient', 'aws-batch'
 
     >>> runner_module_argument("Invalid Name")
     Traceback (most recent call last):
         ...
-    argparse.ArgumentTypeError: invalid runtime name: 'Invalid Name' (normalized to 'invalid-name'); valid names are: 'docker', 'conda', 'ambient', 'aws-batch'
+    argparse.ArgumentTypeError: invalid runtime name: 'Invalid Name' (normalized to 'invalid-name'); valid names are: 'docker', 'conda', 'singularity', 'ambient', 'aws-batch'
     """
     try:
         return runner_module(name)

nextstrain/cli/command/check_setup.py

@@ -1,7 +1,7 @@
 """
 Checks for supported build environments (aka Nextstrain runtimes).
 
-Four runtimes are tested by default:
+Five runtimes are tested by default:
 
   • Our Docker image is the preferred build environment.  Docker itself must
     be installed and configured on your computer first, but once it is, the
@@ -12,6 +12,12 @@
     ambient environment, but is less isolated and robust than the Docker
     runtime.
 
+  • Our Singularity runtime uses the same container image as our Docker
+    runtime.  Singularity must be installed and configured on your computer
+    first, although it is often already present on HPC systems.  This runtime
+    is more isolated and reproducible than the Conda runtime, but potentially
+    less so than the Docker runtime.
+
   • Your ambient environment will be tested for snakemake, augur, and auspice.
     Their presence implies a working build environment, but does not guarantee
     it.

nextstrain/cli/command/shell.py

@@ -9,7 +9,7 @@
 from ..argparse import add_extended_help_flags
 from ..errors import UserError
 from ..paths import SHELL_HISTORY
-from ..runner import docker, conda
+from ..runner import docker, conda, singularity
 from ..util import colored, remove_prefix, runner_name, warn
 from ..volume import store_volume, NamedVolume
 
@@ -37,7 +37,7 @@ def register_parser(subparser):
     runner.register_runners(
         parser,
         exec    = ["bash", ...],
-        runners = [docker, conda])
+        runners = [docker, conda, singularity])
 
     return parser
 
@@ -55,35 +55,35 @@ def run(opts):
 
     overlay_volumes = [v for v in opts.volumes if v is not opts.build]
 
-    if overlay_volumes and opts.__runner__ is not docker:
+    if overlay_volumes and opts.__runner__ not in {docker, singularity}:
         raise UserError(f"""
             The {runner_name(opts.__runner__)} runtime does not support overlays (e.g. of {overlay_volumes[0].name}).
-            Use the Docker runtime (--docker) if overlays are necessary.
+            Use the Docker or Singularity runtimes (via --docker or --singularity) if overlays are necessary.
             """)
 
     print(colored("bold", "Entering the Nextstrain build environment"))
     print()
 
-    if opts.volumes and opts.__runner__ is docker:
+    if opts.volumes and opts.__runner__ in {docker, singularity}:
         print(colored("bold", "Mapped volumes:"))
 
-        # This is more tightly coupled to the Docker runner than I'd like (i.e.
-        # assuming /nextstrain/…), but right now that's the only runner this
-        # command supports (and the only one it makes sense to).
-        #   -trs, 25 Sept 2018
+        # This is more tightly coupled to the Docker/Singularity runners than
+        # I'd like (i.e.  assuming /nextstrain/…), but the number of runtimes
+        # will always be small so some special-casing seems ok.
+        #   -trs, 5 Jan 2023 (updated from 25 Sept 2018)
         for volume in opts.volumes:
-            print("  /nextstrain/%s is from %s" % (volume.name, volume.src.resolve(strict = True)))
+            print("  %s is from %s" % (docker.mount_point(volume), volume.src.resolve(strict = True))) # type: ignore
 
         print()
 
     print(colored("bold", 'Run the command "exit" to leave the build environment.'))
     print()
 
     with resources.as_file("bashrc") as bashrc:
-        # Ensure the history file exists to pass checks the Docker runner
-        # performs for mounted volumes.  This also makes sure that the file is
-        # writable by the Conda runtime too by ensuring the parent directory
-        # exists.
+        # Ensure the history file exists to pass checks the Docker/Singularity
+        # runners perform for mounted volumes.  This also makes sure that the
+        # file is writable by the Conda runtime too by ensuring the parent
+        # directory exists.
         #
         # Don't use strict=True because it's ok if it doesn't exist yet!
         history_file = SHELL_HISTORY.resolve()
@@ -101,7 +101,7 @@ def run(opts):
                 "--rcfile", str(bashrc),
             ]
 
-        elif opts.__runner__ is docker:
+        elif opts.__runner__ in {docker, singularity}:
             opts.volumes.append(NamedVolume("bashrc", bashrc, dir = False, writable = False))
 
             history_volume = NamedVolume("bash_history", history_file, dir = False)

nextstrain/cli/command/update.py

@@ -5,8 +5,8 @@
 without arguments.  Provide a runtime name as an argument to update a specific
 runtime instead.
 
-Only two runtimes currently support updates: Docker and Conda.  Both may take
-several minutes as new software versions are downloaded.
+Three runtimes currently support updates: Docker, Conda, and Singularity.
+Updates may take several minutes as new software versions are downloaded.
 
 This command also checks for newer versions of the Nextstrain CLI (the
 ``nextstrain`` program) itself and will suggest upgrade instructions if an

nextstrain/cli/command/view.py

@@ -55,7 +55,7 @@
 from typing import Iterable, NamedTuple, Tuple, Union
 from .. import runner
 from ..argparse import add_extended_help_flags, SUPPRESS, SKIP_AUTO_DEFAULT_IN_HELP
-from ..runner import docker, ambient, conda
+from ..runner import docker, ambient, conda, singularity
 from ..util import colored, remove_suffix, warn
 from ..volume import NamedVolume
 
@@ -136,7 +136,7 @@ def register_parser(subparser):
     runner.register_runners(
         parser,
         exec    = ["auspice", "view", "--verbose", "--datasetDir=.", "--narrativeDir=."],
-        runners = [docker, ambient, conda])
+        runners = [docker, ambient, conda, singularity])
 
     return parser
 
@@ -176,7 +176,7 @@ def run(opts):
 
     # A volume which will be our working dir.
     working_volume = NamedVolume("auspice/data", data_dir)
-    opts.volumes.append(working_volume) # for Docker
+    opts.volumes.append(working_volume) # for Docker and Singularity
 
     # If auspice/ exists, then use it for datasets.  Otherwise, look for
     # datasets in the given dir.

nextstrain/cli/runner/__init__.py

@@ -5,6 +5,7 @@
 from . import (
     docker as __docker,
     conda as __conda,
+    singularity as __singularity,
     ambient as __ambient,
     aws_batch as __aws_batch,
 )
@@ -36,18 +37,21 @@
 if TYPE_CHECKING and MYPY:
     docker = cast(RunnerModule, __docker)
     conda = cast(RunnerModule, __conda)
+    singularity = cast(RunnerModule, __singularity)
     ambient = cast(RunnerModule, __ambient)
     aws_batch = cast(RunnerModule, __aws_batch)
 else:
     docker = __docker
     conda = __conda
+    singularity = __singularity
     ambient = __ambient
     aws_batch = __aws_batch
 
 
 all_runners: List[RunnerModule] = [
     docker,
     conda,
+    singularity,
     ambient,
     aws_batch,
 ]
@@ -145,10 +149,11 @@ def register_arguments(parser: ArgumentParser, runners: List[RunnerModule], exec
         "development options",
         "These should generally be unnecessary unless you're developing Nextstrain.")
 
-    # Image to use; shared by Docker and AWS Batch runners
+    # Image to use; shared by Docker, AWS Batch, and Singularity runners
     development.add_argument(
         "--image",
-        help    = "Container image name to use for the Nextstrain computing environment",
+        help    = "Container image name to use for the Nextstrain computing environment "                           # type: ignore
+                  f"(default: %(default)s for Docker and AWS Batch, {singularity.DEFAULT_IMAGE} for Singularity)",  # type: ignore
         metavar = "<image>",
         default = docker.DEFAULT_IMAGE) # type: ignore
 
@@ -214,11 +219,16 @@ def run(opts: Options, working_volume: NamedVolume = None, extra_env: Mapping =
 
             If you need the --image option, please select another runner (e.g.
             with the --docker option) that supports it.  Currently --image is
-            supported by the Docker (--docker) and AWS Batch (--aws-batch)
-            runners.  You can check if your setup supports these runners with
-            `nextstrain check-setup`.
+            supported by the Docker (--docker), AWS Batch (--aws-batch), and
+            Singularity (--singularity) runners.  You can check if your setup
+            supports these runners with `nextstrain check-setup`.
             """)
 
+    # Account for potentially different defaults for --image depending on the
+    # selected runner.
+    if opts.__runner__ is singularity and opts.image is docker.DEFAULT_IMAGE: # type: ignore
+        opts.image = singularity.DEFAULT_IMAGE # type: ignore
+
     return opts.__runner__.run(opts, argv, working_volume = working_volume, extra_env = extra_env, cpus = cpus, memory = memory)