manifest subcommand creates invalid manifest list

[dmcgowan]

Currently creating manifest lists from the experimental manifest subcommand uses an incorrect for the manifest, creating invalid manifests. These manifests fail to be pullable with containerd since containerd validates the size. This has lead to broken images being pushed to registries.

See conversation from containerd/containerd#2401

[tonistiigi]

From looking at the code the issue seems to be that the original manifest is not stored in the original form and instead it is reserialized inside a new json object ImageManifest. Even though it uses Payload() function and variables suggests the size is taken over raw data it has actually been remarshalled in the disk already and the original form may be lost.

[thaJeztah]

ping @clnperez @estesp PTAL

[clnperez]

Odd. There was an issue (that I can't find now) about this. And I went to a lot of trouble to get it back to looking exactly the way it looked originally. The issue had to do with the manifest changing (just the tabs in it), and so the hash was different. If anyone has that in their history or inbox please link.

@dmcgowan what version of the cli do you have?

[clnperez]

Ah, hidden in collapsed history. Starting here: hinshun commented on Nov 2, 2017

So it should be fixed in the original, but is not, apparently. I'll take a look.

[vielmetti]

Downstream impact includes a broken CoreDNS for Kubernetes release at kubernetes/kubernetes#65253

which is marked as priority/critical-urgent in that repo.

[flx42]

Answering containerd/containerd#2401 (comment)

@tonistiigi so you're right if you restrict yourself exclusively to code comments of the exported functions/fields. But then, only 3 lines below, this can be confusing:
https://github.com/docker/distribution/blob/749f6afb4572201e3c37325d0ffedb6f32be8950/manifest/schema2/manifest.go#L92-L93

Anyway, the current manifest code seems to need the unmarshalled schema2.DeserializedManifest, in order to perform the blob mount requests to the registry. So carrying an immutable blob of the struct + the struct would be redundant with what's already present indocker/distribution.

[luxas]

cc @dims @mkumatag

[bmichaud]

I am now getting what appears to be the same issue. I am on MacOS 12.3.1
I am working in a colima/lima environment, which is running this version of containerd:
containerd github.com/containerd/containerd v1.5.8 1e5ef943eb76627a6d3b6de8cd1ef6537f393a71

Docker version:

$> docker version
Client: Docker Engine - Community
 Version:           20.10.14
 API version:       1.41
 Go version:        go1.18
 Git commit:        a224086349
 Built:             Fri Mar  4 17:18:07 2022
 OS/Arch:           darwin/amd64
 Context:           colima
 Experimental:      true

Server:
 Engine:
  Version:          20.10.11
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.10
  Git commit:       847da184ad5048b27f5bdf9d53d070f731b43180
  Built:            Fri Nov 19 03:41:34 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.5.8
  GitCommit:        1e5ef943eb76627a6d3b6de8cd1ef6537f393a71
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:

I am getting the following error while building a Docker image from a base node image (node:14.19.1 for building and node:14.19.1-alpine3.15 for runtime)

 => ERROR [stage-1  1/17] FROM docker.io/library/node:14.17-alpine3.14@sha256:7964eefba059e1536cca5c8da0181457b62d32f88a66fa44cb52734cd3b28c29  277.2s
 => => resolve docker.io/library/node:14.17-alpine3.14@sha256:7964eefba059e1536cca5c8da0181457b62d32f88a66fa44cb52734cd3b28c29                    0.0s
 => => sha256:1e738e76d3209f29b5a2a3284864656e933431bb77be4997155954afa813f0e8 2.36MB / 2.36MB                                                    2.9s
 => => sha256:7964eefba059e1536cca5c8da0181457b62d32f88a66fa44cb52734cd3b28c29 1.43kB / 1.43kB                                                    0.0s
 => => sha256:3f4c922d5f29da10fc2a5f1ab6a9812e7aadfc88ecc9e150ef86066f449e55ec 1.16kB / 1.16kB                                                    0.0s
 => => sha256:6c8d3cd2ef0c192a8ec333ca1043574e5e6c04f85265349dac7ec8aaed59e32e 6.53kB / 6.53kB                                                    0.0s
 => => sha256:a0d0a0d46f8b52473982a3c466318f479767577551a53ffc9074c9fa7035982e 2.81MB / 2.81MB                                                    1.7s
#### This one never matches. The download is attempted 3 times ######
 => => sha256:ac0b39f3af6d33c53223dc20d75bc9a69155cdce8bf1eaa84683507cde187863 35.65MB / 36.40MB                                                277.2s
######################
 => => extracting sha256:a0d0a0d46f8b52473982a3c466318f479767577551a53ffc9074c9fa7035982e                                                         0.3s
 => => sha256:064b7e3f032bee1f118f3ee4743c3a4fd09b0180e49e36e3f88839a1a7c5b935 281B / 281B                                                        1.9s
 => [internal] load build context                                                                                                                 0.1s
 => => transferring context: 43.67kB                                                                                                              0.0s
 => CACHED [builder 2/6] WORKDIR /app                                                                                                             0.0s
 => CACHED [builder 3/6] COPY package*.json ./                                                                                                    0.0s
 => [builder 4/6] COPY . .                                                                                                                        0.2s
 => [builder 5/6] RUN npm install                                                                                                                65.4s
 => [builder 6/6] RUN npm run build:prod                                                                                                         98.3s
------
 > [stage-1  1/17] FROM docker.io/library/node:14.17-alpine3.14@sha256:7964eefba059e1536cca5c8da0181457b62d32f88a66fa44cb52734cd3b28c29:
------
failed commit on ref "layer-sha256:ac0b39f3af6d33c53223dc20d75bc9a69155cdce8bf1eaa84683507cde187863": "layer-sha256:ac0b39f3af6d33c53223dc20d75bc9a69155cdce8bf1eaa84683507cde187863" failed size validation: 36397077 != 36398486: failed precondition

I have tried different combinations of the node and alpine versions in the FROM line, but the problem persists. Any idea how to fix this?