push image (with warmed base image(OCI format)) to Dockerhub got MANIFEST_INVALID error

[luxurine]

Actual behavior
When push builded image, with base image(ubuntu:20.04) cached by warmer, will get error:

error pushing image: failed to push to destination luxurine/counter:latest: PUT https://index.docker.io/v2/luxurine/counter/manifests/latest: 
MANIFEST_INVALID: manifest invalid; mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' 
not 'application/vnd.oci.image.manifest.v1+json'

(If there is no cache for base image, image push is ok)

Expected behavior
Image should be pushed successfully.

Preliminary analysis

It looks like ubuntu:20.04 use "mediaType": "application/vnd.oci.image.manifest.v1+json", but dockerhub expect application/vnd.docker.distribution.manifest.v2+json.

The strange thing is, if build without base image cache, pushing image is just ok, I guess reasons could be:

1. if build without base image cache, kaniko will do automatic transformation
2. if build with base image cache, as cache dir is readonly in kaniko executor's view, it will build on top of the cache

kaniko can cache images in a local directory that can be volume mounted into the kaniko pod. To do so, the cache must first be populated, as it is read-only. We provide a kaniko cache warming image at gcr.io/kaniko-project/warmer

Some proposal：

• warmer add flag to control transformation, like OCI --> Docker
• executor support base image cache（assume cache dir is readwrite）（related issue Feature request: Cache base image in repository #2257 ）
• or, executor copy cached base image and do some transformation before build

To Reproduce
Steps to reproduce the behavior:

1. prepare manifests

λ tree .
.
├── Dockerfile
├── bin
│   └── counter
└── config.json

2 directories, 5 files

2. warm base image

$ docker run -v $(pwd):/workspace gcr.io/kaniko-project/warmer:v1.9.1 --cache-dir=/workspace/cache --image="ubuntu:20.04"
INFO[0000] Retrieving image manifest ubuntu:20.04
INFO[0000] Retrieving image ubuntu:20.04 from registry index.docker.io

4. build and push image

$ docker run -ti --rm -v `pwd`:/workspace -v `pwd`/config.json:/kaniko/.docker/config.json:ro gcr.io/kaniko-project/executor:v1.9.1-debug --cache=true --cache-dir=/workspace/cache --dockerfile=Dockerfile --destination=luxurine/counter:latest
INFO[0002] Retrieving image manifest ubuntu:20.04
INFO[0002] Retrieving image ubuntu:20.04 from registry index.docker.io
INFO[0005] Found sha256:3626dff0d616e8ee7065a9ac8c7117e904a4178725385910eeecd7f1872fc12d in local cache
INFO[0005] Found manifest at /workspace/cache/sha256:3626dff0d616e8ee7065a9ac8c7117e904a4178725385910eeecd7f1872fc12d.json
INFO[0005] Built cross stage deps: map[]
INFO[0005] Retrieving image manifest ubuntu:20.04
INFO[0005] Returning cached image manifest
INFO[0005] Found sha256:3626dff0d616e8ee7065a9ac8c7117e904a4178725385910eeecd7f1872fc12d in local cache
INFO[0005] Found manifest at /workspace/cache/sha256:3626dff0d616e8ee7065a9ac8c7117e904a4178725385910eeecd7f1872fc12d.json
INFO[0005] Executing 0 build triggers
INFO[0005] Building stage 'ubuntu:20.04' [idx: '0', base-idx: '-1']
INFO[0005] Unpacking rootfs as cmd COPY bin/ /app requires it.
INFO[0007] COPY bin/ /app
INFO[0007] Taking snapshot of files...
INFO[0007] WORKDIR /app
INFO[0007] Cmd: workdir
INFO[0007] Changed working directory to /app
INFO[0007] No files changed in this command, skipping snapshotting.
INFO[0007] ENTRYPOINT ["/app/counter"]
INFO[0007] No files changed in this command, skipping snapshotting.
INFO[0007] Pushing image to luxurine/counter:latest
error pushing image: failed to push to destination luxurine/counter:latest: PUT https://index.docker.io/v2/luxurine/counter/manifests/latest: MANIFEST_INVALID: manifest invalid; mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not 'application/vnd.oci.image.manifest.v1+json'

Additional Information

• Dockerfile

FROM ubuntu:20.04

COPY bin/ /app

WORKDIR /app

ENTRYPOINT ["/app/counter"]

• Kaniko Image: gcr.io/kaniko-project/executor:v1.9.1-debug

ref:

• https://docs.docker.com/registry/spec/manifest-v2-2/
• https://github.com/opencontainers/image-spec/blob/main/manifest.md
• docker manifest inspect doesn't support OCI manifests moby/moby#44898

[jamgregory]

I've just experienced a similar issue building a local development image and pushing it to our own GitLab Registry instance (albeit without the useful error message - I just got MANIFEST_INVALID: manifest invalid; map[]

I'm not sure how you worked around this @luxurine, but my solution was to build the Ubuntu base image myself:

# Based on: https://git.launchpad.net/cloud-images/+oci/ubuntu-base/tree/?h=focal-20.04
FROM scratch
ADD source/ubuntu-focal-oci-amd64-root.tar.gz /

[SeWieland]

I think this is related to #1836

[aaron-prindle]

I believe this is a dupe of #1836 which was recently fixed and in the latest kaniko release (v1.15.0). Closing, feel free to re-open if there is more to address here

[luxurine]

I've just tried kaniko-project/executor:v1.17.0 again, still the same problem. Could you please reopen this issue? @aaron-prindle

manifest

# Dockfile
FROM ubuntu:20.04 # with OCI format
WORKDIR /app

step

1. warmup image

$ docker run -v $(pwd):/workspace gcr.io/kaniko-project/warmer:v1.17.0 --cache-dir=/workspace/cache --image=ubuntu:20.04

# output
INFO[0000] Retrieving image manifest ubuntu:20.04
INFO[0000] Retrieving image ubuntu:20.04 from registry index.docker.io

2. build and push

$ docker run -ti --rm -v `pwd`:/workspace -v `pwd`/config.json:/kaniko/.docker/config.json:ro gcr.io/kaniko-project/executor:v1.17.0-debug --cache=true --cache-dir=/workspace/cache --dockerfile=Dockerfile --destination=luxurine/demo:latest --verbosity=debug

# output
INFO[0002] Retrieving image manifest ubuntu:20.04
INFO[0002] Retrieving image ubuntu:20.04 from registry index.docker.io
INFO[0006] Found sha256:218bb51abbd1864df8be26166f847547b3851a89999ca7bfceb85ca9b5d2e95d in local cache
INFO[0006] Found manifest at /workspace/cache/sha256:218bb51abbd1864df8be26166f847547b3851a89999ca7bfceb85ca9b5d2e95d.json
INFO[0006] Built cross stage deps: map[]
INFO[0006] Retrieving image manifest ubuntu:20.04
INFO[0006] Returning cached image manifest
INFO[0006] Found sha256:218bb51abbd1864df8be26166f847547b3851a89999ca7bfceb85ca9b5d2e95d in local cache
INFO[0006] Found manifest at /workspace/cache/sha256:218bb51abbd1864df8be26166f847547b3851a89999ca7bfceb85ca9b5d2e95d.json
INFO[0006] Executing 0 build triggers
...
INFO[0006] Pushing image to luxurine/demo:latest
error pushing image: failed to push to destination luxurine/demo:latest: PUT https://index.docker.io/v2/luxurine/demo/manifests/latest: MANIFEST_INVALID: manifest invalid; mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not 'application/vnd.oci.image.manifest.v1+json'

[tanadeau]

I've also seen this with 1.18.0 and 1.19.0. Seems possibly related to newer issue #2883. We're using GitLab Container Registry and not AWS ECR.

[hojo0590]

maybe related/a regression of #2713?

[mhio]

Ran into this issue on executor-v1.23.2 and recent warmer-latest (doesn't report a version) to a gitlab v16.11.5 container registry.

[luxurine]

Ran into this issue on executor-v1.23.2 and recent warmer-latest (doesn't report a version) to a gitlab v16.11.5 container registry.

@mhio here is a temporary fix you can try:

1. fork kaniko repo
2. update func func (c *cachedImage) Manifest() (*v1.Manifest, error) in pkg/cache/cache.go and return tabball's manifest by default.
3. build image

Warning

target image is always in docker format

func (c *cachedImage) Manifest() (*v1.Manifest, error) {
	if c.mfst == nil {
		return c.Image.Manifest()
	}
	return c.mfst, nil
}

->

func (c *cachedImage) Manifest() (*v1.Manifest, error) {
	mfst, err := c.Image.Manifest()
	if mfst != nil {
		return mfst, err
	}
	return c.mfst, nil
}

[mhio]

Thanks @luxurine we'll give that a try, disabling the warmed cache was the quick workaround.