G_A_C in GKE Tekton pipeline: open /secret/kaniko-secret: no such file or directory #3828

yuwenma · 2020-03-13T21:57:14Z

Expected behavior

From the example, when running skaffold in a tekton pipeline using GKE cluster, an env var GOOGLE_APPLICATION_CREDENTIALS needs to be specified with a kaniko secret mounted to it.

Either by running "skaffold generate-pipeline" or using the examples from the examples folder should successfully mount the secret to the GOOGLE_APPLICATION_CREDENTIALS.

Actual behavior

The image gcr.io/k8s-skaffold/skaffold (tested v1.4.0 and v1.5.0) can't find the secret from GOOGLE_APPLICATION_CREDENTIALS. Here's the error message:

Creating kaniko secret [<MY NAMESPACE>/kaniko-secret]...
Building [gcr.io/XXX/redisslave]...
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "gcr.io/<MY PROJECT>/<MY IMG>:7ef7eb8": creating push check transport for gcr.io failed: Get https://gcr.io/v2/token?scope=repository%3A<MYPROJECT>%2F<MY IMG>%3Apush%2Cpull&service=gcr.io: invoking docker-credential-gcr: exit status 1; output: docker-credential-gcr/helper: could not retrieve GCR's access token: google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable: open /secret/kaniko-secret: no such file or directory
time="2020-03-13T21:27:24Z" level=fatal msg="build failed: building [gcr.io/yuwenma-gke-playground/redisslave]: waiting for pod to complete: condition error: pod already in terminal phase: Failed"

Information

Skaffold version: v1.4.0
Operating system: OSX/Linux
Contents of skaffold.yaml:

apiVersion: skaffold/v2alpha4
kind: Config
metadata:
  name: ci-test-
build:
  artifacts:
  - image: gcr.io/k8s-skaffold/skaffold
profiles:
- name: oncluster
  build:
    artifacts:
    - image: gcr.io/<MY PROJECT>/<My IMG>
      context: .
      kaniko: {}
    tagPolicy:
      gitCommit: {}
    cluster:
      pullSecretName: kaniko-secret
      namespace: <MY NAMESPACE>

Here's my tekton task config

apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: skaffold-build-and-push
  namespace: <MY NAMESPACE>
spec:
  inputs:
    resources:
    - name: workspace
      type: git
    params:
    - name: pathToSkaffold
      default: /workspace/workspace/skaffold.yaml # my input resource is located in /workspace/workspace
  outputs:
    resources:
    - name: builtImage
      type: image
  steps:
  - name: run-skaffold-build
    image: gcr.io/k8s-skaffold/skaffold:v1.5.0 # also tested v1.4.0
    command:
    - skaffold
    - build
    args:
    - --filename
    - $(inputs.params.pathToSkaffold) # location of my skaffold.yaml 
    - --profile
    - oncluster 
    - --file-output
    - build.out
    workingDir: /workspace/workspace
    resources: {}
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: /secret/kaniko-secret  # tried both kaniko-secret and kaniko-secret.json (the file name contained in the k8s kaniko-secret). The skaffold/examples folder seems to have both formats.
    volumeMounts:
    - name: kaniko-secret
      mountPath: /secret
  volumes:
  - name: kaniko-secret
    secret:
      secretName: kaniko-secret

Additional Information

The kaniko-secret itself is correct. The service account of the kaniko-secret has the expected access to upload images to my gcr and I've tested the secret using kaniko directly in tekton task config.

steps:
- name: build-and-push
  image: gcr.io/kaniko-project/executor:v0.15.0
  command:
  - /kaniko/executor
  args:
  - --dockerfile=$(inputs.params.pathToDockerFile)
  - --destination=$(outputs.resources.builtImage.url)
  - --context=$(inputs.params.pathToContext)
  env:
  - name: GOOGLE_APPLICATION_CREDENTIALS
    value: /secret/kaniko-secret.json
  volumeMounts:
  - name: kaniko-secret
    mountPath: /secret
volumes:
- name: kaniko-secret
  secret:
    secretName: kaniko-secret

The text was updated successfully, but these errors were encountered:

yuwenma · 2020-03-13T21:58:12Z

/kind bug

tstromberg · 2020-04-20T20:57:34Z

Something is definitely broken here, but we haven't yet had the time to reproduce the issue.

Lowering the priority due to current bandwidth.

tstromberg · 2020-04-28T16:59:48Z

@yuwenma - Hey Yuwen. I'd like to take a closer look at this issue.

I'm new to the project, so I'm having a difficult time following along. Can you help me clarify:

Is one of the existing examples broken? If so, which example, and with what skaffold command-line arguments?
If additional files are required for reproduction, do you mind creating an example GitHub repo that I can checkout? Please provide exact command-lines or a script to reproduce.

Thank you for your help!

yuwenma · 2020-04-28T20:17:28Z

@tstromberg Sure thing!

here is the broken example. Two things are incorrect after I triaged the issue:

The image gcr.io/k8s-skaffold/skaffold:test-version doesn't exist in the container registry https://console.cloud.google.com/gcr/images/k8s-skaffold/GLOBAL/skaffold?tag=test-version&gcrImageListsize=30
When using the official images (tried both tags v1.4.0 and v1.5.0), the error message is

Creating kaniko secret [<MY NAMESPACE>/kaniko-secret]...
Building [gcr.io/XXX/redisslave]...
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "gcr.io/<MY PROJECT>/<MY IMG>:7ef7eb8": creating push check transport for gcr.io failed: Get https://gcr.io/v2/token?scope=repository%3A<MYPROJECT>%2F<MY IMG>%3Apush%2Cpull&service=gcr.io: invoking docker-credential-gcr: exit status 1; output: docker-credential-gcr/helper: could not retrieve GCR's access token: google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable: open /secret/kaniko-secret: no such file or directory
time="2020-03-13T21:27:24Z" level=fatal msg="build failed: building [gcr.io/yuwenma-gke-playground/redisslave]: waiting for pod to complete: condition error: pod already in terminal phase: Failed"

You can see from the example's volume mount config, the k8s secret kaniko-secret is in fact mounted to the correct path.

You can find the full tekton config from the above issue description. Let me know if there's anything unclear.

chanseokoh · 2020-04-28T22:59:53Z

So I got curious about this issue and decided to test it out myself. I don't know how Skaffold handles all these, but here's what I did to make this work.

(Optional) forget setting GOOGLE_APPLICATION_CREDENTIALS. (I suspect Skaffold sets GOOGLE_APPLICATION_CREDENTIALS=/secret/kaniko-secret.)

 workingDir: /workspace/workspace
 resources: {}
-env:
-- name: GOOGLE_APPLICATION_CREDENTIALS
-  value: /secret/kaniko-secret  # tried both kaniko-secret and kaniko-secret.json (the file name contained in the k8s kaniko-secret). The skaffold/examples folder seems to have both formats.
-volumeMounts:
-- name: kaniko-secret

UPDATE: volume mounts are unnecessary too. Remove them.

Create the secret kaniko-secret with the data key name kaniko-secret. @yuwenma based on what you said, I am certain that you created the secret with the name kaniko-secret.json.

[run-build] Creating kaniko secret [default/kaniko-secret]...
[run-build] Building [gcr.io/< ... redacted ... >/skaffold-pipeline-test]...
[run-build] INFO[0000] Resolved base name alpine:3.10 to alpine:3.10 
[run-build] INFO[0000] Resolved base name alpine:3.10 to alpine:3.10 
[run-build] INFO[0000] Downloading base image alpine:3.10           
[run-build] ERROR: logging before flag.Parse: E0428 22:49:12.200734       1 metadata.go:142] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
[run-build] ERROR: logging before flag.Parse: E0428 22:49:12.203472       1 metadata.go:159] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
[run-build] INFO[0000] Error while retrieving image from cache: getting file info: stat /cache/sha256:a143f3ba578f79e2c7b3022c488e6e12a35836cd4a6eb9e363d7f3a07d848590: no such file or directory 
[run-build] INFO[0000] Downloading base image alpine:3.10           
[run-build] INFO[0001] Built cross stage deps: map[]                
[run-build] INFO[0001] Downloading base image alpine:3.10           
[run-build] INFO[0001] Error while retrieving image from cache: getting file info: stat /cache/sha256:a143f3ba578f79e2c7b3022c488e6e12a35836cd4a6eb9e363d7f3a07d848590: no such file or directory 
[run-build] INFO[0001] Downloading base image alpine:3.10           
[run-build] INFO[0001] Skipping unpacking as no commands require it. 
[run-build] INFO[0001] Taking snapshot of full filesystem...        
[run-build] INFO[0001] USER 1000                                    
[run-build] INFO[0001] cmd: USER                                    
[run-build] time="2020-04-28T22:49:17Z" level=warning msg="error adding artifacts to cache; caching may not work as expected: inspecting image: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"

yuwenma · 2020-04-29T19:24:15Z

So I got curious about this issue and decided to test it out myself. I don't know how Skaffold handles all these, but here's what I did to make this work.

Thanks @chanseokoh for jumping in. Just to confirm a couple of things, the config I post is tekton which is different from running an isolated docker task.

Besides, the GOOGLE_APPLICATION_CREDENTIALS env is specifically required by kaniko when pushing the images to a container registry. If it's not GKE cluster and not container registry, then it may not be required.

Besides, hard coding the secret as "kaniko-secret" won't fit our needs. because what I do is giving our customers the option to set up kaniko-secret themselves and eventually this part would be changed to something different. @tstromberg

chanseokoh · 2020-04-29T20:07:48Z

Thanks @chanseokoh for jumping in. Just to confirm a couple of things, the config I post is tekton which is different from running an isolated docker task.

I ran a Tekton task.

Besides, the GOOGLE_APPLICATION_CREDENTIALS env is specifically required by kaniko when pushing the images to a container registry. If it's not GKE cluster and not container registry, then it may not be required.

I am saying GOOGLE_APPLICATION_CREDENTIALS seems to be overwritten by Skaffold (Skaffold folks need to confirm) so whatever you set in the Tekton Task spec becomes irrelevant.

Besides, hard coding the secret as "kaniko-secret" won't fit our needs. because what I do is giving our customers the option to set up kaniko-secret themselves and eventually this part would be changed to something different.

Not sure what you mean by "hard-coding" here. Just like what you did, I am providing kaniko-secert as stored as a k8s secret. You just need to name the key as kaniko-secret rather than kaniko-secret.json.

chanseokoh · 2020-04-29T23:40:58Z

Did a bit more testing, and actually, it turns out you don't even need to mount a secret. You just need to set the data key name of the secret to kaniko-secret.

     - --profile
     - oncluster 
     - --file-output
     - build.out
     workingDir: /workspace/workspace
     resources: {}
-    env:
-    - name: GOOGLE_APPLICATION_CREDENTIALS
-      value: /secret/kaniko-secret  # tried both kaniko-secret and kaniko-secret.json (the file name contained in the k8s kaniko-secret). The skaffold/examples folder seems to have both formats.
-    volumeMounts:
-    - name: kaniko-secret
-      mountPath: /secret
-  volumes:
-  - name: kaniko-secret
-    secret:
-      secretName: kaniko-secret

I think Skaffold does all the magic with cluster: pullSecretName:.

    cluster:
      pullSecretName: kaniko-secret

When there is no secret:

[run-build]  - gcr.io/.../skaffold-pipeline-test: Not found. Building
[run-build] Creating kaniko secret [default/kaniko-secret]...
[run-build] time="2020-04-29T23:31:36Z" level=fatal msg="build failed: setting up pull secret: checking for existing kaniko secret: secrets \"kaniko-secret\" not found"

When the secret data key name does not match:

[run-build] Building [gcr.io/.../skaffold-pipeline-test]...
[run-build] error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "gcr.io/.../skaffold-pipeline-test:79a9a6b": creating push check transport for gcr.io failed: Get https://gcr.io/v2/token?scope=repository%3A...%2Fskaffold-pipeline-test%3Apush%2Cpull&service=gcr.io: invoking docker-credential-gcr: exit status 1; output: docker-credential-gcr/helper: could not retrieve GCR's access token: google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable: open /secret/kaniko-secret: no such file or directory
[run-build] time="2020-04-29T23:36:25Z" level=fatal msg="build failed: building [gcr.io/.../skaffold-pipeline-test]: pod has failed"

When providing a bad secret:

[run-build] Building [gcr.io/.../skaffold-pipeline-test]...
[run-build] error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "gcr.io/.../skaffold-pipeline-test:79a9a6b": creating push check transport for gcr.io failed: Get https://gcr.io/v2/token?scope=repository%3A...%2Fskaffold-pipeline-test%3Apus
h%2Cpull&service=gcr.io: invoking docker-credential-gcr: exit status 1; output: docker-credential-gcr/helper: could not retrieve GCR's access token: oauth2: cannot fetch token: 400 Bad Request
[run-build] Response: {                                                                                                                                                                             
[run-build]   "error": "invalid_grant",                                                                                                                                                             
[run-build]   "error_description": "Bad Request"
[run-build] }                                                                                     
[run-build] time="2020-04-29T23:31:18Z" level=fatal msg="build failed: building [gcr.io/.../skaffold-pipeline-test]: pod has failed"

When providing a working secret, the same output as I reported eariler:

[run-build] Building [gcr.io/.../skaffold-pipeline-test]...
[run-build] INFO[0000] Resolved base name alpine:3.10 to alpine:3.10 
[run-build] INFO[0000] Resolved base name alpine:3.10 to alpine:3.10 
[run-build] INFO[0000] Downloading base image alpine:3.10           
[run-build] ERROR: logging before flag.Parse: E0429 23:34:55.911503       1 metadata.go:159] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
[run-build] ERROR: logging before flag.Parse: E0429 23:34:55.917240       1 metadata.go:142] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
[run-build] INFO[0000] Error while retrieving image from cache: getting file info: stat /cache/sha256:a143f3ba578f79e2c7b3022c488e6e12a35836cd4a6eb9e363d7f3a07d848590: no such file or directory 
[run-build] INFO[0000] Downloading base image alpine:3.10           
[run-build] INFO[0001] Built cross stage deps: map[]                
[run-build] INFO[0001] Downloading base image alpine:3.10           
[run-build] INFO[0001] Error while retrieving image from cache: getting file info: stat /cache/sha256:a143f3ba578f79e2c7b3022c488e6e12a35836cd4a6eb9e363d7f3a07d848590: no such file or directory 
[run-build] INFO[0001] Downloading base image alpine:3.10           
[run-build] INFO[0001] Skipping unpacking as no commands require it. 
[run-build] INFO[0001] Taking snapshot of full filesystem...        
[run-build] INFO[0001] USER 1000                                    
[run-build] INFO[0001] cmd: USER                                    
[run-build] time="2020-04-29T23:35:00Z" level=warning msg="error adding artifacts to cache; caching may not work as expected: inspecting image: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"

yuwenma · 2020-05-09T08:30:55Z

Thanks Chanseoko, I did another try but still couldn't get it work. Here's what I did:

First of all, my original post is based on skaffold resource v2alpha4. I updated the resource to v2beta3. Then confirmed my skaffold.yaml (see below) works by running skaffold build --profile oncluster locally (the skaffold CLI is also updated to the latest v1.9.1). Images are build and pushed to the expected place successfully.

apiVersion: skaffold/v2beta3
kind: Config
metadata: 
  name: app-cicd-demo-
profiles:
  - name: oncluster
    build:
      artifacts:
        - image: gcr.io/yuwen-cicd-demo2/ci-redisslave1
          context: redis-slave

Then I updated the tekton config as you suggested (no G_A_C env), updated theskaffold container image from v1.15 to the latest, and also updated tekton from V1alpha1 to v1beta1 for reliability.

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: build-and-push
  namespace: "cicd" 
spec:
  params:
  - name: pathToDockerFile
    default: /workspace/workspace/php-redis/Dockerfile
    description: The path to the dockerfile to build
  - name: pathToContext
    default: /workspace/workspace
    description: The build context used by Kaniko
    default: skaffold.yaml
  steps:
  - name: run-skaffold-build
    image: gcr.io/k8s-skaffold/skaffold
    command:
    - skaffold
    - build
    args:
    - --filename
    - $(params.skaffoldConfig) # location of my skaffold.yaml 
    - --profile
    - oncluster 
    - --file-output
    - build.out
    workingDir: /workspace/workspace
    resources: {}

Then the tekton task reports gcloud 400 errors

 kubectl -n cicd logs ci-run-vpzgd-build-frontend-nmmkh-pod-9s6sb -c step-run-skaffold-build
Generating tags...
 - gcr.io/yuwen-cicd-demo2/ci-redisslave1 -> gcr.io/yuwen-cicd-demo2/ci-redisslave1:4b3da59
Checking cache...
 - gcr.io/yuwen-cicd-demo2/ci-redisslave1: Not found. Building
Building [gcr.io/yuwen-cicd-demo2/ci-redisslave1]...
ERROR: gcloud crashed (MetadataServerException): HTTP Error 400: Bad Request

If you would like to report this issue, please run the following command:
  gcloud feedback

To check gcloud for common problems, please run the following command:
  gcloud info --run-diagnostics
couldn't build "gcr.io/yuwen-cicd-demo2/ci-redisslave1": docker build: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

I also tried including the G_A_C env, and got the same error.

@chanseokoh Let me know if there's any other info you may want to know. Thanks for jumping in and giving suggestions! Would be great to hear some insights from @tstromberg and @nkubala as well!

chanseokoh · 2020-05-11T20:23:37Z

@yuwenma here's a minimal, complete step-by-step example that I created for demonstration. Note I'm just telling you how you can make this work; I do think Skaffold has room for improvement or at least needs to fix the sample with better documentation around providing the Kaniko secret.

Preparation

Go to Cloud Console and create a GKE cluster with the latest Master version (1.15.11-gke.12 as of now).
Install latest Tekton following the official instructions.
1. Grant cluster-admin permissions to your Google account.
```
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole=cluster-admin \
--user=$(gcloud config get-value core/account)
```
2. Install latest Tekton:
  kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
3. Wait until all the components are in the Running state, as explained in the official install guide.

Bind the default pod account (system:serviceaccount:default:default) with cluster-admin.

kubectl create clusterrolebinding default-account-cluster-admin-binding \
--clusterrole=cluster-admin \
--user=system:serviceaccount:default:default

Working Sample

$ git clone https://github.com/chanseokoh/skaffold-generate-pipeline-test
$ cd skaffold-generate-pipeline-test

First, update skaffold.yaml to push to your repository. (This is your original skaffold.yaml.)

 - name: oncluster
   build:
     artifacts:
-    - image: gcr.io/chanseok-playground-new/skaffold-pipeline-test
+    - image: gcr.io/yuwen-cicd-demo2/skaffold-pipeline-test
       context: .
       kaniko: {}
     tagPolicy:

Let's try with a wrong secret that will fail.

$ kubectl apply -f wrong_secret.yaml  # WRONG SECRET
$ kubectl apply -f run.yaml

Watch it fails due to the wrong secret.

[git-source-source-hfcv8] {"level":"info","ts":1589226111.9015768,"caller":"git/git.go:136","msg":"Successfully cloned https://github.com/chanseokoh/skaffold-generate-pipeline-test.git @ 4f73a2cbd3fa0b6557b360cd6aa44c4e3ee55f4c (grafted, HEAD, origin/master) in path /workspace/source"}
[git-source-source-hfcv8] {"level":"info","ts":1589226111.9404137,"caller":"git/git.go:177","msg":"Successfully initialized and updated submodules in path /workspace/source"}

[run-build] Generating tags...
[run-build]  - gcr.io/chanseok-playground-new/skaffold-pipeline-test -> gcr.io/chanseok-playground-new/skaffold-pipeline-test:4f73a2c
[run-build] Checking cache...
[run-build]  - gcr.io/chanseok-playground-new/skaffold-pipeline-test: Not found. Building
[run-build] Creating kaniko secret [default/cool-secret]...
[run-build] Building [gcr.io/chanseok-playground-new/skaffold-pipeline-test]...
[run-build] E0511 19:42:02.942617       1 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
[run-build]     For verbose messaging see aws.Config.CredentialsChainVerboseErrors
[run-build] error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "gcr.io/chanseok-playground-new/skaffold-pipeline-test:4f73a2c": resolving authorization for gcr.io failed: error getting credentials - err: exit status 1, out: `docker-credential-gcr/helper: could not retrieve GCR's access token: google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable: invalid character 'W' looking for beginning of value`
[run-build] couldn't build "gcr.io/chanseok-playground-new/skaffold-pipeline-test": pod has failed

At this point, it is already proved that it picks up the very wrong secret I provided. No such error you observed as open /secret/kaniko-secret: no such file or directory. The secret file is there, and it is being used.

Next, test with your working secret.

$ kubectl delete secret/cool-secret  # DELETE THE WRONG SECRET
$ kubectl create secret generic cool-secret --from-file=kaniko-secret=my-service-account-key.json  # USE YOUR KEY FILE
$ kubectl apply -f run.yaml

Watch the image is built and pushed to gcr.io/chanseok-playground-new/skaffold-pipeline-test:4f73a2c. Kaniko runs my Dockerfile.

[run-build] Generating tags...
[run-build]  - gcr.io/chanseok-playground-new/skaffold-pipeline-test -> gcr.io/chanseok-playground-new/skaffold-pipeline-test:4f73a2c
...
[run-build] INFO[0001] Resolved base name golang:1.12.9-alpine3.10 to builder
[run-build] INFO[0001] Retrieving image manifest golang:1.12.9-alpine3.10
...
[run-build] INFO[0001] Retrieving image manifest golang:1.12.9-alpine3.10
[run-build] INFO[0001] Retrieving image manifest alpine:3.10
[run-build] INFO[0001] Retrieving image manifest alpine:3.10
[run-build] INFO[0002] Built cross stage deps: map[0:[/app]]
[run-build] INFO[0002] Retrieving image manifest golang:1.12.9-alpine3.10
[run-build] INFO[0002] Retrieving image manifest golang:1.12.9-alpine3.10
[run-build] INFO[0002] Executing 0 build triggers
[run-build] INFO[0002] Unpacking rootfs as cmd COPY main.go . requires it.
[run-build] INFO[0008] COPY main.go .
[run-build] INFO[0008] Resolving 1 paths
[run-build] INFO[0008] Taking snapshot of files...
[run-build] INFO[0008] RUN go build -o /app main.go

For example,

$ docker inspect gcr.io/chanseok-playground-new/skaffold-pipeline-test:4f73a2c
[]
Error: No such object: gcr.io/chanseok-playground-new/skaffold-pipeline-test:4f73a2c
$ docker pull gcr.io/chanseok-playground-new/skaffold-pipeline-test:4f73a2c
4f73a2c: Pulling from chanseok-playground-new/skaffold-pipeline-test
21c83c524219: Pull complete 
a5bbe7b5cd60: Pull complete 
Digest: sha256:cbff90a252dbbccc506e83d21404de11ea4a2e7f47eddf4b05109bf19f843e3c
Status: Downloaded newer image for gcr.io/chanseok-playground-new/skaffold-pipeline-test:4f73a2c
gcr.io/chanseok-playground-new/skaffold-pipeline-test:4f73a2c
$ docker run --rm gcr.io/chanseok-playground-new/skaffold-pipeline-test:4f73a2c
Hello world!
Hello world!
Hello world!
...

tejal29 · 2020-05-11T21:08:28Z

@yuwenma skaffold hardcoded the GOOGLE_ APPLICATION_CREDENTAILS to /<volumeMountPath>/kaniko-secret.

If the secret you created was at different filepath, kaniko pod would fail.
This was not documented well.

With this change,

You can now use, pullSecret in skaffold config to point to the secret filepath within your kaniko secret.

yuwenma · 2020-05-12T17:25:02Z

Thanks @tejal29 I tried changing the kaniko-secret to match the hard-coded one and it works.
However, I encountered a new issue. Here's the error code

 - gcr.io/yuwen-cicd-demo2/ci-redisslave1: Not found. Building
Building [gcr.io/yuwen-cicd-demo2/ci-redisslave1]...
couldn't build "gcr.io/yuwen-cicd-demo2/ci-redisslave1": undefined artifact type: {DockerArtifact:<nil> BazelArtifact:<nil> JibArtifact:<nil> KanikoArtifact:0xc0003150e0 BuildpackArtifact:<nil> CustomArtifact:<nil>}

I did specified an artifact KainkoArtifact in my skaffold.yaml

apiVersion: skaffold/v2beta3
kind: Config
metadata: 
  name: app-cicd-demo-
profiles:
  - name: oncluster
    build:
      artifacts:
        - image: gcr.io/yuwen-cicd-demo2/ci-redisslave1
          context: redis-slave
          kaniko: 
            dockerfile: /workspace/workspace/redis-slave/Dockerfile

Wondering it complains that the artiact type is not defined. @nkubala @tejal29

tstromberg · 2020-05-12T20:19:28Z

@yuwenma - It's tough to tell from this error, but I think this may be caused because there is no cluster specified to build on, so it's falling back onto the local builder, which doesn't support Kaniko.

skaffold/pkg/skaffold/build/local/local.go

Line 104 in eab0d15

return "", fmt.Errorf("undefined artifact type: %+v", artifact.ArtifactType)

For example, see examples/kaniko/skaffold.yaml

build:
  artifacts:
    - image: skaffold-example
      kaniko:
        cache: {}
  cluster:
    pullSecretName: e2esecret
    namespace: default

yuwenma · 2020-05-12T21:15:22Z

Is there a way to specify the cluster name? From the examples/generate-pipeline, it seems that the skaffold image doesn't need the cluster to be specified.
https://github.com/GoogleContainerTools/skaffold/blob/master/examples/generate-pipeline/expectedPipeline.yaml#L31

nkubala · 2020-06-08T20:26:02Z

@yuwenma I think this issue was resolved right? #4161 is tracking the other issue that you ran into in the process.

yuwenma · 2020-06-08T20:34:36Z

Yes, feel free to close the issue @nkubala

yuwenma changed the title ~~[bug] GOOGLE_APPLICATION_CREDENTIALS raises credential errors~~ GOOGLE_APPLICATION_CREDENTIALS can't be mounted to skaffold image in GKE cluster Mar 13, 2020

nkubala added area/tekton build/kaniko kind/bug Something isn't working priority/p1 High impact feature/bug. labels Mar 19, 2020

tstromberg added priority/p2 May take a couple of releases needs-reproduction needs reproduction from the maintainers to validate the issue is truly a skaffold bug and removed priority/p1 High impact feature/bug. labels Apr 20, 2020

tstromberg changed the title ~~GOOGLE_APPLICATION_CREDENTIALS can't be mounted to skaffold image in GKE cluster~~ GOOGLE_APPLICATION_CREDENTIALS in a tekton pipeline: open /secret/kaniko-secret: no such file or directory Apr 28, 2020

tstromberg changed the title ~~GOOGLE_APPLICATION_CREDENTIALS in a tekton pipeline: open /secret/kaniko-secret: no such file or directory~~ G_A_C in GKE Tekton pipeline: open /secret/kaniko-secret: no such file or directory Apr 28, 2020

chanseokoh mentioned this issue May 1, 2020

Docker Credential GoogleContainerTools/jib#2440

Closed

nkubala added the fixit label May 11, 2020

nkubala assigned tejal29 May 11, 2020

tejal29 mentioned this issue May 11, 2020

Use pullSecretPath to set GOOGLE_APPLICATION_CREDENTIALS #4147

Merged

tstromberg mentioned this issue May 12, 2020

undefined artifact type: {DockerArtifact:<nil> BazelArtifact:<nil> JibArtifact:<nil> KanikoArtifact:0xc0003150e0 BuildpackArtifact:<nil> CustomArtifact:<nil>} #4161

Closed

tstromberg removed the needs-reproduction needs reproduction from the maintainers to validate the issue is truly a skaffold bug label May 18, 2020

tejal29 closed this as completed in #4147 Jun 10, 2020

chanseokoh mentioned this issue Jun 23, 2020

add docs on how to define kaniko volume mount and update secret instructions #4362

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

G_A_C in GKE Tekton pipeline: open /secret/kaniko-secret: no such file or directory #3828

G_A_C in GKE Tekton pipeline: open /secret/kaniko-secret: no such file or directory #3828

yuwenma commented Mar 13, 2020 •

edited by tstromberg

Loading

yuwenma commented Mar 13, 2020

tstromberg commented Apr 20, 2020

tstromberg commented Apr 28, 2020

yuwenma commented Apr 28, 2020

chanseokoh commented Apr 28, 2020 •

edited

Loading

yuwenma commented Apr 29, 2020 •

edited

Loading

chanseokoh commented Apr 29, 2020 •

edited

Loading

chanseokoh commented Apr 29, 2020

yuwenma commented May 9, 2020

chanseokoh commented May 11, 2020 •

edited

Loading

tejal29 commented May 11, 2020 •

edited

Loading

yuwenma commented May 12, 2020

tstromberg commented May 12, 2020 •

edited

Loading

yuwenma commented May 12, 2020

nkubala commented Jun 8, 2020

yuwenma commented Jun 8, 2020

G_A_C in GKE Tekton pipeline: open /secret/kaniko-secret: no such file or directory #3828

G_A_C in GKE Tekton pipeline: open /secret/kaniko-secret: no such file or directory #3828

Comments

yuwenma commented Mar 13, 2020 • edited by tstromberg Loading

Expected behavior

Actual behavior

Information

Additional Information

yuwenma commented Mar 13, 2020

tstromberg commented Apr 20, 2020

tstromberg commented Apr 28, 2020

yuwenma commented Apr 28, 2020

chanseokoh commented Apr 28, 2020 • edited Loading

yuwenma commented Apr 29, 2020 • edited Loading

chanseokoh commented Apr 29, 2020 • edited Loading

chanseokoh commented Apr 29, 2020

yuwenma commented May 9, 2020

chanseokoh commented May 11, 2020 • edited Loading

Preparation

Working Sample

tejal29 commented May 11, 2020 • edited Loading

yuwenma commented May 12, 2020

tstromberg commented May 12, 2020 • edited Loading

yuwenma commented May 12, 2020

nkubala commented Jun 8, 2020

yuwenma commented Jun 8, 2020

yuwenma commented Mar 13, 2020 •

edited by tstromberg

Loading

chanseokoh commented Apr 28, 2020 •

edited

Loading

yuwenma commented Apr 29, 2020 •

edited

Loading

chanseokoh commented Apr 29, 2020 •

edited

Loading

chanseokoh commented May 11, 2020 •

edited

Loading

tejal29 commented May 11, 2020 •

edited

Loading

tstromberg commented May 12, 2020 •

edited

Loading