Support squashfs->tar in push of an OCI-SIF data container #2874
Milestone
Comments
3 tasks
dtrudg
added a commit
to dtrudg/singularity
that referenced
this issue
May 23, 2024
Add a new command, `singularity data package`, that will create an OCI-SIF data container that 'packages' a host file or directory. The OCI-SIF data container consists of an OCI image, with a custom config mediaType. At present it will always hold a single layer, in squashfs format, containing the file / directory that was packaged. We currently use a flow that: * Reads the file / directory into a tar stream. * Presents the tar stream as a ggcr layer. * Mutates the layer into squashfs format. * Writes the image containing the mutated layer into an OCI-SIF. Arguably, given we are producing OCI-SIF with only a squashfs layer, we could directly call out to mksquashfs and then insert the layer from that - without going through a tar reader. We might adopt this optimisation in future, but for now we follow similar patterns to OCI-SIF container image handling, where we start with a tar layer and mutate. We probably need to consider what we do with layer format on push / pull, e.g. sylabs#2874 so let's keep tar in mind for now. Note that the command is using `singularity package <src> <dst>` argument order. This is different from our `build <dst> <src>`... but is generally more intuitive? Thoughts? Closes sylabs#2872
dtrudg
added a commit
to dtrudg/singularity
that referenced
this issue
May 23, 2024
Add a new command, `singularity data package`, that will create an OCI-SIF data container that 'packages' a host file or directory. The OCI-SIF data container consists of an OCI image, with a custom config mediaType. At present it will always hold a single layer, in squashfs format, containing the file / directory that was packaged. We currently use a flow that: * Reads the file / directory into a tar stream. * Presents the tar stream as a ggcr layer. * Mutates the layer into squashfs format. * Writes the image containing the mutated layer into an OCI-SIF. Arguably, given we are producing OCI-SIF with only a squashfs layer, we could directly call out to mksquashfs and then insert the layer from that - without going through a tar reader. We might adopt this optimisation in future, but for now we follow similar patterns to OCI-SIF container image handling, where we start with a tar layer and mutate. We probably need to consider what we do with layer format on push / pull, e.g. sylabs#2874 so let's keep tar in mind for now. Note that the command is using `singularity package <src> <dst>` argument order. This is different from our `build <dst> <src>`... but is generally more intuitive? Thoughts? Closes sylabs#2872
dtrudg
added a commit
to dtrudg/singularity
that referenced
this issue
May 23, 2024
Add a new command, `singularity data package`, that will create an OCI-SIF data container that 'packages' a host file or directory. The OCI-SIF data container consists of an OCI image, with a custom config mediaType. At present it will always hold a single layer, in squashfs format, containing the file / directory that was packaged. We currently use a flow that: * Reads the file / directory into a tar stream. * Presents the tar stream as a ggcr layer. * Mutates the layer into squashfs format. * Writes the image containing the mutated layer into an OCI-SIF. Arguably, given we are producing OCI-SIF with only a squashfs layer, we could directly call out to mksquashfs and then insert the layer from that - without going through a tar reader. We might adopt this optimisation in future, but for now we follow similar patterns to OCI-SIF container image handling, where we start with a tar layer and mutate. We probably need to consider what we do with layer format on push / pull, e.g. sylabs#2874 so let's keep tar in mind for now. Note that the command is using `singularity package <src> <dst>` argument order. This is different from our `build <dst> <src>`... but is generally more intuitive? Thoughts? Closes sylabs#2872
dtrudg
added a commit
to dtrudg/singularity
that referenced
this issue
May 23, 2024
Add a new command, `singularity data package`, that will create an OCI-SIF data container that 'packages' a host file or directory. The OCI-SIF data container consists of an OCI image, with a custom config mediaType. At present it will always hold a single layer, in squashfs format, containing the file / directory that was packaged. We currently use a flow that: * Reads the file / directory into a tar stream. * Presents the tar stream as a ggcr layer. * Mutates the layer into squashfs format. * Writes the image containing the mutated layer into an OCI-SIF. Arguably, given we are producing OCI-SIF with only a squashfs layer, we could directly call out to mksquashfs and then insert the layer from that - without going through a tar reader. We might adopt this optimisation in future, but for now we follow similar patterns to OCI-SIF container image handling, where we start with a tar layer and mutate. We probably need to consider what we do with layer format on push / pull, e.g. sylabs#2874 so let's keep tar in mind for now. Note that the command is using `singularity package <src> <dst>` argument order. This is different from our `build <dst> <src>`... but is generally more intuitive? Thoughts? Closes sylabs#2872
dtrudg
added a commit
to dtrudg/singularity
that referenced
this issue
May 23, 2024
Add a new command, `singularity data package`, that will create an OCI-SIF data container that 'packages' a host file or directory. The OCI-SIF data container consists of an OCI image, with a custom config mediaType. At present it will always hold a single layer, in squashfs format, containing the file / directory that was packaged. We currently use a flow that: * Reads the file / directory into a tar stream. * Presents the tar stream as a ggcr layer. * Mutates the layer into squashfs format. * Writes the image containing the mutated layer into an OCI-SIF. Arguably, given we are producing OCI-SIF with only a squashfs layer, we could directly call out to mksquashfs and then insert the layer from that - without going through a tar reader. We might adopt this optimisation in future, but for now we follow similar patterns to OCI-SIF container image handling, where we start with a tar layer and mutate. We probably need to consider what we do with layer format on push / pull, e.g. sylabs#2874 so let's keep tar in mind for now. Note that the command is using `singularity package <src> <dst>` argument order. This is different from our `build <dst> <src>`... but is generally more intuitive? Thoughts? Closes sylabs#2872
cyanezstange
pushed a commit
to cyanezstange/singularity
that referenced
this issue
Jun 4, 2024
Add a new command, `singularity data package`, that will create an OCI-SIF data container that 'packages' a host file or directory. The OCI-SIF data container consists of an OCI image, with a custom config mediaType. At present it will always hold a single layer, in squashfs format, containing the file / directory that was packaged. We currently use a flow that: * Reads the file / directory into a tar stream. * Presents the tar stream as a ggcr layer. * Mutates the layer into squashfs format. * Writes the image containing the mutated layer into an OCI-SIF. Arguably, given we are producing OCI-SIF with only a squashfs layer, we could directly call out to mksquashfs and then insert the layer from that - without going through a tar reader. We might adopt this optimisation in future, but for now we follow similar patterns to OCI-SIF container image handling, where we start with a tar layer and mutate. We probably need to consider what we do with layer format on push / pull, e.g. sylabs#2874 so let's keep tar in mind for now. Note that the command is using `singularity package <src> <dst>` argument order. This is different from our `build <dst> <src>`... but is generally more intuitive? Thoughts? Closes sylabs#2872
|
This already works... |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When an OCI-SIF data container (#2872) is pushed then the data held in squashfs format must be converted to a tar layer if requested at push-time (#2848)
The text was updated successfully, but these errors were encountered: