Old Docker images incompatible with Singularity v4.0.3

[stxue1]

Something seemed to have changed around Singularity v4 that prevents Singularity from pulling images that are too old:

$ singularity --version
singularity-ce version 4.0.3-jammy
$ singularity --verbose run docker://quay.io/cmarkello/samtools_picard@sha256:e484603c61e1753c349410f0901a7ba43a2e5eb1c6ce9a240b7f737bba661eb4
FATAL:   Unable to handle docker://quay.io/cmarkello/samtools_picard@sha256:e484603c61e1753c349410f0901a7ba43a2e5eb1c6ce9a240b7f737bba661eb4 uri: failed to get checksum for docker://quay.io/cmarkello/samtools_picard@sha256:e484603c61e1753c349410f0901a7ba43a2e5eb1c6ce9a240b7f737bba661eb4: not a valid image manifest or image index

It looks like downloading and reuploading the image to quay updates the image and resolves this issue:

$ singularity --verbose run docker://quay.io/adamnovak/samtools_picard:latest # works

[stxue1]

I'm unable to run the giraffe test for Toil locally due to this with Singularity v4.1.2

[2024-07-23T17:36:02-0700] [MainThread] [C] [toil.worker] Worker crashed with traceback:
Traceback (most recent call last):
  File "/home/heaucques/Documents/toil/src/toil/worker.py", line 439, in workerScript
    job._runner(jobGraph=None, jobStore=job_store, fileStore=fileStore, defer=defer)
  File "/home/heaucques/Documents/toil/src/toil/job.py", line 2984, in _runner
    returnValues = self._run(jobGraph=None, fileStore=fileStore)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/src/toil/job.py", line 2895, in _run
    return self.run(fileStore)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/src/toil/wdl/wdltoil.py", line 145, in decorated
    return decoratee(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/src/toil/wdl/wdltoil.py", line 2267, in run
    task_container._pull(miniwdl_logger, cleanup)
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/WDL/runtime/backend/cli_subprocess.py", line 222, in _pull
    raise DownloadFailed(image) from None
WDL.runtime.error.DownloadFailed: unable to download /home/heaucques/.cache/miniwdl/docker___quay.io_glennhickey_pigz_2.3.1.sif

Which is due to this line:

vg_wdl/tasks/bioinfo_utils.wdl

Line 141 in 00167b2

docker: "quay.io/glennhickey/pigz:2.3.1"

The image at https://quay.io/repository/glennhickey/pigz?tab=tags&tag=2.3.1 is also using the old manifest

singularity pull /home/heaucques/.cache/miniwdl/docker___quay.io_glennhickey_pigz_2.3.1.sif docker://quay.io/glennhickey/pigz:2.3.1
FATAL:   While making image from oci registry: error fetching image to cache: failed to get checksum for docker://quay.io/glennhickey/pigz:2.3.1: not a valid image manifest or image index

[stxue1]

This is related to docker/roadmap#662

I've tried using crane to automatically convert from manifest v1 to v2, but apparently the manifest is even older than what crane can support: google/go-containerregistry#377

Error: unsupported MediaType: "application/vnd.docker.distribution.manifest.v1+prettyjws", see https://github.com/google/go-containerregistry/issues/377

Pulling and pushing the image with an older version of Docker seems to be the only option; since I originally posted this, I believe schema 1 support has been removed.