When --cache true and --registry-mirror ... kaniko cannot find manifest

[ajcollett]

Actual behavior
When running the docker kaniko debug image with --cache true and --registry-mirror ...
kaniko throws error about not being able to find the manifest of the base image.

This works if library/ is added to the base image name in the FROM line, in the Dockerfile. This shouldn't be required?

Error output:

INFO[0000] Resolved base name ubuntu:xenial to ubuntu:xenial 
 INFO[0000] Resolved base name ubuntu:xenial to ubuntu:xenial 
 INFO[0000] Retrieving image manifest ubuntu:xenial      
 ERRO[0000] Error while retrieving image from cache: ubuntu:xenial MANIFEST_UNKNOWN: manifest unknown; map[Tag:xenial] 
 INFO[0000] Retrieving image manifest ubuntu:xenial      
 error building image: MANIFEST_UNKNOWN: manifest unknown; map[Tag:xenial]

Expected behavior
Docker image should be built assuming library/ for the base image, unless otherwise specified.

To Reproduce
Steps to reproduce the behavior:

1. Set both --cache true and --registry-mirror ... on a build
2. Build an image who's base FROM line does NOT specify library/ in the image name.
3. Build fails with manifest unknown.

Additional Information
Failed gitlab pipeline:
https://git.cs.sun.ac.za/Computer-Science/docker-prebuild/sun-ubuntu/-/jobs/226756

Working pipeline after adding library/... to dockerfile FROM line.
https://git.cs.sun.ac.za/Computer-Science/docker-prebuild/sun-ubuntu/-/jobs/226767

Example repo with dockerfile and .gitlab-ci.yml:
https://git.cs.sun.ac.za/Computer-Science/docker-prebuild/sun-ubuntu

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[tejal29]

--registry-mirror was a new feature added recently in https://github.com/GoogleContainerTools/kaniko/pull/836/files
@greut would like to take a look at this?

[greut]

@tejal29 I'll give it a look, cheers.

[greut]

@ajcollett I suspect that's because the mirror is using the strict validation name, which adds library.

Do you mind trying this image which uses weak validation?
https://hub.docker.com/layers/greut/executor/debug-0.22.1-alpha/images/sha256-5e03d7eb4260574f57e781e5bbb06240a51e105286a40acde80a6bd695813e21?context=explore

If this doesn't work, the cache will have to use strict validation as well.

[zaventh]

@greut

This does not resolve the issue for me:

...
 Using Kubernetes executor with image greut/executor:debug-0.22.1-alpha ...
...
 INFO[0041] Using dockerignore file: /builds/[redacted]/proxy/.dockerignore 
 INFO[0041] Retrieving image manifest node:12-alpine     
 ERRO[0041] Error while retrieving image from cache: node:12-alpine GET https://[redacted-mirror]/v2/node/manifests/12-alpine: MANIFEST_UNKNOWN: manifest unknown; map[Tag:12-alpine] 
 INFO[0041] Retrieving image manifest node:12-alpine     
 error building image: GET https://[redacted-mirror]/v2/node/manifests/12-alpine: MANIFEST_UNKNOWN: manifest unknown; map[Tag:12-alpine]

The first line of the Dockerfile is FROM node:12-alpine

[greut]

@zaventh okay, I'll try to setup the cache for ourselves. The alternative I've had in mind what that the cache should use the StrictValidation.

[ajcollett]

Hello @zaventh,

Just want to clarify what you mean by below?

@ajcollett I suspect that's because the mirror is using the strict validation name, which adds library.

I setup the docker "pull-through cache" myself. I can post the docker file if that'd be helpful, but it's very standard. Normally we set the "registry-mirrors" settings up in the daemon on our servers, and the image names just work. No library needed.

Pretty much use this: https://docs.docker.com/registry/recipes/mirror/

[greut]

does it work when cache is off?

[greut]

Another question, do you mind pointing me where the cache-dir is populated by the warmer?

https://github.com/GoogleContainerTools/kaniko#caching-base-images

[greut]

okay, I can reproduce it.

$ docker run --rm -v $(pwd)/test:/workspace -v $(pwd)/cache:/cache greut/executor:debug-v0.22.0 -f Dockerfile.3 --registry-mirror [REDACTED] --no-push
INFO[0000] Retrieving image manifest ubuntu:bionic      
ERRO[0003] Error while retrieving image from cache: ubuntu:bionic GET https://[REDACTED]/v2/ubuntu/manifests/bionic: MANIFEST_UNKNOWN: manifest unknown; map[Tag:bionic] 
INFO[0003] Retrieving image manifest ubuntu:bionic      
error building image: GET https://[REDACTED]/v2/ubuntu/manifests/bionic: MANIFEST_UNKNOWN: manifest unknown; map[Tag:bionic]

[greut]

@ajcollett greut/executor:debug-v0.22.1-alpha.1 #1264

[zaventh]

@greut That fix does indeed work. I was able to build images where both the FROM is a base image like node:12-alpine as well as from private registries. It also works in multi-stage Docker builds. Thanks. 💯

[ajcollett]

@greut It works for me now! I have tested both with and without library/.

Thanks for this!

Some more info. I am using a GitLab instance, which has built in capability to cache layers. My .gitlab-ci.yml for interest sake (sorry, I didn't twig this might be helpful earlier):

.build-kaniko-image:
  stage: build
  variables:
    BUILD_DOCKERFILE: $CI_PROJECT_DIR/Dockerfile
    IMAGE_TAG: $CI_COMMIT_TAG
    REGISTRY_MIRROR: "our-internal-mirror"
    CACHE: "true"
    CLEANUP: "true"
  image:
    name: greut/executor:debug-v0.22.1-alpha.1
    entrypoint: [""]
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --registry-mirror ${REGISTRY_MIRROR} --context ${CI_PROJECT_DIR} --dockerfile ${BUILD_DOCKERFILE} --destination ${CI_REGISTRY_IMAGE}:${IMAGE_TAG} --cache=${CACHE} --cleanup=${CLEANUP}
  tags:
    - our-docker-image-builder