Support --bind of images in --oci mode (from SIF)

[dtrudg]

In native mode, a --bind source can be a SIF image, in which case filesystem partitions contained within the image can be bound into a container.

This functionality should be supported in --oci mode.

Some consideration is required of the oci-sif layout. What exactly to we want to support binding out of a SIF into a container? Would we allow e.g. binding a specific layer of an oci-sif image etc.

---

From conversation with @tri-adam and @wobito the following seems apparent:

--bind from images is generally used to provide reference data to an application container at runtime. E.g. a specific reference genomic sequence to an RNA-Seq alignment tool. The advantage of being able to store the reference data in a SIF is primarily that the same infrastructure can be used to distribute data as application containers.

In Singularity's existing native mode, --bind from a SIF will mount the first (or optionally a specified) descriptor that is marked as data in a SIF file. There are then two scenarios for the structure of the SIF(s) involved:

1. A separate SIF file, distinct from the SIF holding an application container, is used to provide data that will be mounted.
2. A data partition is added to a SIF that already holds an application container.

Scenario 2 is somewhat confusing. It's easy to understand 1.. where a SIF provides data, and is intended to be used in conjunction with one or more SIFs containing applications. In scenario 2, data embedded in the same SIF as an application container might logically be assumed to be relevant only for that application, but it is actually possible to bind the data onto a completely separate application container, run from a different SIF. Knowing when data is provided in an application container, and how to use it, is also non-obvious.

Since overlays in OCI-SIF will be supported via #1480 - let's consider only Scenario 1 - separate OCI-SIFs holding data, to be used in conjunction with an OCI-SIF holding the application container.

• It should be possible to create an OCI-SIF holding data only (no application), in a simple manner. It should be easier than the current instructions for native SIF data partitions that use cryptic siftool commands (https://docs.sylabs.io/guides/4.1/user-guide/bind_paths_and_mounts.html#sif-image-files)

• The data will be stored in a manner that is similar to an OCI application container image, i.e. with a manifest structure defining the layout of the content.

• Data in an OCI-SIF will be read-only at this time. read/write is most often sought for overlays and is tackled in Support embedded overlay in --oci mode (from SIF) #1480. Avoiding r/w data binds from OCI-SIF will avoid the issue of manifest/data consistency at this time.

• The OCI-style layout will allow push/pull to/form OCI registries in a similar manner to OCI application container images. The blob content of the OCI-SIF is transferred... instead of the OCI-SIF itself as a single opaque blob. An oras:// push may still allow the entire SIF to be transferred if necessary.

• The existing --bind and --mount syntax for images will be followed.

---

• OCI-SIF data container creation command #2872
• Support OCI-SIF data-container in --bind & --mount #2873
• Support squashfs->tar in push of an OCI-SIF data container #2874