"no valid providers in chain" in Jenkins

[juanmacasau]

Im trying to use Kaniko in my Jenkinsfile:

def label = "kaniko-${UUID.randomUUID().toString()}"

podTemplate(name: 'kaniko', label: label, yaml: """
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:debug
    imagePullPolicy: Always
    command:
    - /busybox/cat
    tty: true
    volumeMounts:
      - name: jenkins-docker-cfg
        mountPath: /kaniko/.docker
  volumes:
  - name: jenkins-docker-cfg
    projected:
      sources:
      - secret:
          name: regcred
          items:
            - key: .dockerconfigjson
              path: config.json
""") { 
[....]
  node(label) {  
    stage('Dockerize') {
        container(name: 'kaniko') {
            unstash 'build'
            sh 'ls'
            sh '''
            /kaniko/executor -f `pwd`/Dockerfile -c `pwd` --insecure --skip-tls-verify --cache=true --verbosity=trace --destination=index.docker.io/USERNAME/REPOSITORY:TAG
            
            '''
        }
      }
    }

And always I get the following error

�[37mDEBU�[0m[0000] Copying file /home/jenkins/agent/workspace/contagio/PRO/api-sintomas-pipeline/Dockerfile to /kaniko/Dockerfile 
�[37mTRAC�[0m[0000] Adding /var/run to initialWhitelist     
E0516 19:12:00.571357      31 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

And my pipeline does not stop never.

Somebody knows what is the problem?

Thanks!

[juanmacasau]

Somebody can help me? It is for my final degree project :(

[will-darkwolf]

I am also running into this issue with a gitlab-ci build running on kubernetes.

My gitlab-ci.yml job looks like:

kaniko:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --cache=true --dockerfile "${CI_PROJECT_DIR}/Dockerfile" --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG --context "dir://${CI_PROJECT_DIR}/"

[will-darkwolf]

@juanmacasau

I was able to build with this base image gcr.io/kaniko-project/executor@sha256:f652f28537fa76e8f4f9393de13a064f0206003c451ce2ad6e4359fd5a21acbc from a month ago. Not really a fix, but a workaround.

[ball-hayden]

I'm also seeing this from within a GKE cluster.

We've been using kaniko in the cluster happily. This morning we enabled the workload identity feature and metadata server and kaniko now hangs as described.

[ball-hayden]

In my case, this was caused by a network policy preventing access to the Google Metadata Server (metadata.google.internal).

Even though the network policy allowed egress on port 80, this didn't work.
Only workaround so far is to remove the egress rules from the network policy (so that any connections are allowed).

Kaniko should IMHO timeout after some (reasonably short) period when trying to get credentials.

[tejal29]

duplicate of #1287

[DanielJonesEB]

This is bonkers - why should my Kaniko build need to be able to talk to the Google Metadata Server when it's building from a Dockerfile in a directory, and outputting to a tar file?

[tejal29]

we did a fix in #1445 to address checking pushing permissions when --no-push was specified.
It unfortunately was broken in this follow up commit

Its fixed on master now due to
#1471

This change is pending release.

[andy-paine]

I've tried building the container from HEAD and I'm still seeing some issues unfortunately. I've done a bunch of digging and I think our issue is actually just the "infinite loop" behaviour of the GCR credential package in https://github.com/vdemeester/k8s-pkg-credentialprovider as we are a) running on GCE but b) have blocked access to the metadata server as we are running untrusted workloads. I think this assumption then doesn't work for our setup https://github.com/vdemeester/k8s-pkg-credentialprovider/blob/8edca87436b8a9a0924e8c6168177cd1854bd186/gcp/metadata.go#L215. Do you think it would be reasonable to either a) not infinitely loop or b) be able to control which providers Kaniko attempts to register (rather than just relying on init functions)?

This is unrelated to the --no-push flag as it is about trying to authenticate with the providers for pulling images in FROM steps.

Thanks for looking into this!

[tejal29]

Thanks @andy-paine for confirming this is related to the GCP metadata server.
I asked internally this question and i haven't got any response yet. I will keep nudging the right ppl.

Thanks
Tejal

[tejal29]

I've tried building the container from HEAD and I'm still seeing some issues unfortunately. I've done a bunch of digging and I think our issue is actually just the "infinite loop" behaviour of the GCR credential package in https://github.com/vdemeester/k8s-pkg-credentialprovider as we are a) running on GCE but b) have blocked access to the metadata server as we are running untrusted workloads. I think this assumption then doesn't work for our setup https://github.com/vdemeester/k8s-pkg-credentialprovider/blob/8edca87436b8a9a0924e8c6168177cd1854bd186/gcp/metadata.go#L215. Do you think it would be reasonable to either a) not infinitely loop or b) be able to control which providers Kaniko attempts to register (rather than just relying on init functions)?

Thanks, i am looking to code where k8Chains.Enabled() is called to see if adding flags can help or disable retrying.