remove redundant intermediate images in build --oci flow
#2308
Comments
|
This is somewhat difficult... To convert the OCI tar to OCI-SIF we first ensure we have the image in an OCI layout, from which it's easy to perform mutations etc. We 'fetch' the tar file into a layout using ggcr. Within the ggcr code the tar will be opened multiple times to retrieve different files. It's not possible to work on a tar stream that cannot be re-opened from the start. We need to directly extract the tar into the layout to bypass the tar temp file. |
|
It's not that difficult... the buildkit tar is an OCI archive rather than a legacy Docker archive these days, so we can simply extract it and we have a layout. |
|
It appears that after #2935 then the better approach is to keep the tarball, but read from it without extracting to a layout directory. This can be acheived by treating it as a docker-archive, rather than an oci-archive. The buildkit tar is an OCI layout, but with the additional docker-specific manifest.json, so ggcr can read from it directly as a tarball. |
dtrudg
changed the title
build --oci flowbuild --oci flow
With sylabs#2935 in place we can now translate a Docker archive tarball into an OCI-SIF without creating a redundant copy of the image as a temporary OCI layout. We've previously been processing the buildkit tarball as an oci-archive, which still requires extraction to a temporary layout, as ggcr's tarball package doesn't accept an archived layout without a manifest.json. Because the buildkit output tarball includes the Docker specific manifest.json, in addition to the standard OCI layout files, it can be treated as a docker-archive - now avoiding the need for the temporary layout. Significantly reduces disk space needed by `build --oci`, and the associated I/O. Fixes sylabs#2308
With sylabs#2935 in place we can now translate a Docker archive tarball into an OCI-SIF without creating a redundant copy of the image as a temporary OCI layout. We've previously been processing the buildkit tarball as an oci-archive, which still requires extraction to a temporary layout, as ggcr's tarball package doesn't accept an archived layout without a manifest.json. Because the buildkit output tarball includes the Docker specific manifest.json, in addition to the standard OCI layout files, it can be treated as a docker-archive - now avoiding the need for the temporary layout. Significantly reduces disk space needed by `build --oci`, and the associated I/O. Fixes sylabs#2308
With sylabs#2935 in place we can now translate a Docker archive tarball into an OCI-SIF without creating a redundant copy of the image as a temporary OCI layout. We've previously been processing the buildkit tarball as an oci-archive, which still requires extraction to a temporary layout, as ggcr's tarball package doesn't accept an archived layout without a manifest.json. Because the buildkit output tarball includes the Docker specific manifest.json, in addition to the standard OCI layout files, it can be treated as a docker-archive - now avoiding the need for the temporary layout. Significantly reduces disk space needed by `build --oci`, and the associated I/O. Fixes sylabs#2308
With sylabs#2935 in place we can now translate a Docker archive tarball into an OCI-SIF without creating a redundant copy of the image as a temporary OCI layout. We've previously been processing the buildkit tarball as an oci-archive, which still requires extraction to a temporary layout, as ggcr's tarball package doesn't accept an archived layout without a manifest.json. Because the buildkit output tarball includes the Docker specific manifest.json, in addition to the standard OCI layout files, it can be treated as a docker-archive - now avoiding the need for the temporary layout. Significantly reduces disk space needed by `build --oci`, and the associated I/O. Fixes sylabs#2308
Currently, building an OCI-SIF using buildkit first builds a tar file of the OCI image, and then converts that tar to an OCI-SIF by treating it as a source for
oci-archive:URI.We should see if we can pipe the output of buildkit straight into OCI-SIF through a pipe, to avoid the creation of the intermediate tar file.
The text was updated successfully, but these errors were encountered: