Broken test: test_manifest.py::test_manifest_platform_variant

[LewisGaul]

Test fails in CI with:

ERROR tests/python_on_whales/components/test_manifest.py::test_manifest_platform_variant - python_on_whales.exceptions.DockerException: The docker command executed was `/usr/bin/docker image pull --quiet arm64v8/busybox:1.35`.
It returned with code 1
The content of stdout is ''
The content of stderr is 'no matching manifest for linux/amd64 in the manifest list entries

Also seen running manually with docker and podman:

$docker pull arm64v8/busybox:1.35
1.35: Pulling from arm64v8/busybox
no matching manifest for linux/amd64 in the manifest list entries

$podman pull arm64v8/busybox:1.35
✔ docker.io/arm64v8/busybox:1.35
Trying to pull docker.io/arm64v8/busybox:1.35...
Error: choosing an image from manifest list docker://arm64v8/busybox:1.35: no image found in image index for architecture amd64, variant "", OS linux

[LewisGaul]

This testcase was added only a couple of months ago in #492, although it passed in the CI at the time. I'm not sure exactly what's changed here to cause the breakage though, perhaps the docker API/registry no longer supports pulling images that are for an architecture not supported by the host? @d4nj1 any thoughts?

Note two of the other testcases in that file were also marked to be skipped in CI back in May...

[d4nj1]

Hi @LewisGaul ,
could it be that the CI system changed from arm64 to amd64? At least the error message complains about platform not matching. There are two things you could test:

1. Force downloading the arm64 busybox image anyways withdocker pull --platform linux/arm64 arm64v8/busybox:1.35
2. Switch to amd64 by changing the pull command to docker pull busybox:1.35

Does this help?

[d4nj1]

Hi @LewisGaul ,
I looked a bit deeper and I think two things come together here.

1. Docker pull refuses to pull images for other platforms if they are not specified with --platform=PLATFORM.
   Output of docker pull --help

Usage:  docker pull [OPTIONS] NAME[:TAG|@DIGEST]

Download an image from a registry

Aliases:
  docker image pull, docker pull

Options:
  -a, --all-tags                Download all tagged images in the repository
      --disable-content-trust   Skip image verification (default true)
      --platform string         Set platform if server is multi-platform capable
  -q, --quiet                   Suppress verbose output

Sadly python-on-whales does not support this at the moment. Only quiet option is implemented, see here.

2. The busybox arm64v8 image does not contain the variant property anymore as it seems to be the only one existing. We could switch to arm32 which still includes the variant property but then again the above mentioned has to be implemented first.

Conclusion:
Easiest way to proceed would be to remove the variant test as it's currently broken anyways. Ideally the functionality from 1. should be implemented.

[d4nj1]

I just added a PR with a fix. Let's wait for review.

[LewisGaul]

@d4nj1 thanks so much for helping out! I'll give it a few days in case @gabrieldemarmiesse wants to take a look but otherwise I can approve/merge to get the CI green again :)

[gabrieldemarmiesse]

Many thanks for the help, I'll take a look this afternoon :)