Unsuccessful cred copy: ".docker" at end of successful build

[gokula-krishna-dev]

Expected Behavior

Exit builds free of warnings

Actual Behavior

At the end of the build I get the warning
[image-digest-exporter-rmrl8] 2020/10/16 06:13:34 unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied

Steps to Reproduce the Problem

1. minikube start --memory 6144 --cpus 2
2. kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
3. Sample creds kubectl create secret docker-registry regcred --docker-server=https://docker.io/v1/ --docker-username=sample --docker-password=123 --docker-email=hi@gmail.com
4. kubectl apply -f sample.yaml where contents are

apiVersion: tekton.dev/v1alpha1
kind: PipelineResource
metadata:
  name: skaffold-git
spec:
  type: git
  params:
    - name: revision
      value: master
    - name: url
      value: https://github.com/GoogleContainerTools/skaffold 
---
apiVersion: tekton.dev/v1alpha1
kind: PipelineResource
metadata:
  name: skaffold-image-leeroy-web
spec:
  type: image
  params:
    - name: url
      value: index.docker.io/sample/bot
---
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: build-docker-image-from-git-source
spec:
  params:
    - name: pathToDockerFile
      type: string
      description: The path to the dockerfile to build
      default: $(resources.inputs.docker-source.path)/Dockerfile
    - name: pathToContext
      type: string
      description: |
        The build context used by Kaniko
        (https://github.com/GoogleContainerTools/kaniko#kaniko-build-contexts)
      default: $(resources.inputs.docker-source.path)
  resources:
    inputs:
      - name: docker-source
        type: git
    outputs:
      - name: builtImage
        type: image
  steps:
    - name: build-and-push
      image: gcr.io/kaniko-project/executor:v0.16.0
      # specifying DOCKER_CONFIG is required to allow kaniko to detect docker credential
      env:
        - name: "DOCKER_CONFIG"
          value: "/tekton/home/.docker/"
      command:
        - /kaniko/executor
      args:
        - --dockerfile=$(params.pathToDockerFile)
        - --destination=$(resources.outputs.builtImage.url)
        - --context=$(params.pathToContext)
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tutorial-service
secrets:
  - name: regcred
---
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: build-docker-image-from-git-source-task-run
spec:
  serviceAccountName: tutorial-service
  taskRef:
    name: build-docker-image-from-git-source
  params:
    - name: pathToDockerFile
      value: Dockerfile
    - name: pathToContext
      value: $(resources.inputs.docker-source.path)/examples/microservices/leeroy-web #configure: may change according to your source
  resources:
    inputs:
      - name: docker-source
        resourceRef:
          name: skaffold-git
    outputs:
      - name: builtImage
        resourceRef:
          name: skaffold-image-leeroy-web

5. The build is succeeded but the the above warning appears in the logs.

Additional Info

• Kubernetes version:

  Output of kubectl version:

Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:43:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

• Tekton Pipeline version:

  Output of tkn version or kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'

Pipeline version: v0.17.1

I couldn't find relevant docs for this warning. Is there any configuration step that I'm missing? or is it ok to ignore the error?

[ghost]

@gokula-krishna-dev thanks for the report - could you share the pod yaml too? kubectl get pod -o yaml <name-of-taskrun-pod> That would be helpful in trying to figure out what happened. Remember to sanitize the YAML if it includes any sensitive info before posting it here.

This message is only a warning but it can be indicative of a problem - did the docker image push succeed?

The message should only appear if the /tekton/home/.docker directory was already created when the Step ran. My hunch right now is that the git PipelineResource injected a Step that ran with a different UID to your Task's Steps. There have been a number of attempts recently to try and get Tekton's supporting containers onto non-root base images and I am guessing that might have played a role here.

[ghost]

Oh, sorry I just realised from your original message that it's the image-digest-exporter step that's raising the error.

So this is likely what appears to be the problem: the image digest exporter injected step is based on a non-root image. The git-init Step and your Task's Step both initialize credentials in /tekton/home/.docker as the root user. Then the image-digest-exporter comes along and tries to initialize the docker credentials too. This fails because the image digest exporter can't overwrite a root-owned directory.

[ghost]

I've been able to reproduce the issue from your example YAMLS and regcred, thanks a lot for those! This should not interfere with the operation of your Task and is a warning related to the difference in UIDs of the various Steps injected into your Task by PipelineResources. I'm going to document the warning message more clearly in #3395 and am going to spend some time trying to figure out if there's a way to suppress these warnings in images like the image-digest-exporter where we know for sure that the credentials don't need to be initialized.

[gokula-krishna-dev]

@sbwsg Noted and thank you for your response. I'll proceed with my setup. Also, I'll be waiting for the fix to be released 😄

[zhangtbj]

Hi @sbwsg ,

Thanks for the info in doc: https://github.com/tektoncd/pipeline/blob/master/docs/auth.md#unsuccessful-cred-copy-warning

And I would like to have a try to set disable-creds-init to "true", but it doesn't work for me, I can still see the warning:

2020/11/30 05:04:36 unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied
{"level":"info","ts":1606712739.351053,"logger":"fallback-logger","caller":"imagedigestexporter/main.go:59","msg":"No index.json found for: image","commit":"0375523"}

I am using the Tekton v0.17.1, is it a new introduced flag or did I set something wrong?

apiVersion: v1
data:
  disable-affinity-assistant: "false"
  disable-creds-init: "true"
  disable-home-env-overwrite: "false"
  disable-working-directory-overwrite: "false"
  require-git-ssh-secret-known-hosts: "false"
  running-in-environment-with-injected-sidecars: "true"
kind: ConfigMap
metadata:
  name: feature-flags
  namespace: tekton-pipelines

Thanks!

[ghost]

@zhangtbj the feature flag was only added in 0.18.0 unfortunately :S

[zhangtbj]

I see, yes I tried the v0.18.0 the feature flag works for me. Thanks!

[georgettica]

could this be applied to a specific task and not to the whole cluster?