[🐛] snyk/kubernetes-monitor GCP Container Registry

[oleksandrs-adorama]

• kubernetes-monitor version [e.g. v2.4.13]
• Cloud runtime [GKE]

Expected behaviour

Authenticate to private container registries should works.

Actual behaviour

kubernetes-monitor Pod cannot pull image for scan and has errors

{"name":"kubernetes-monitor","hostname":"snyk-kubernetes-monitor-8fdcf4ccc-mh4ls","pid":7,"level":40,"message":"WARNING: Could not setup log file in /srv/app/.config/gcloud/logs, (OSError: [Errno 30] Read-only file system: '/srv/app/.config/gcloud'.\nThe configuration directory may not be writable. To learn more, see https://cloud.google.com/sdk/docs/configurations#creating_a_configuration\nERROR: gcloud crashed (OSError): [Errno 30] Read-only file system: '/srv/app/.config/gcloud'\n\nIf you would like to report this issue, please run the following command:\n  gcloud feedback\n\nTo check gcloud for common problems, please run the following command:\n  gcloud info --run-diagnostics\ntime=\"2023-10-06T09:37:10Z\" level=fatal msg=\"initializing source docker://gcr.io/****/****@sha256:62a442d3d2cf1d72758994e66767f2f6dbc8d2e4b5d9886d393509725d5222f2: getting username and password: 1 error occurred:\\n\\t* error getting credentials - err: exit status 1, out: ``\\n\\n\"\n","bin":"skopeo","loggableArguments":["copy","--dest-compress-level","6","docker://gcr.io/****/****@sha256:62a442d3d2cf1d72758994e66767f2f6dbc8d2e4b5d9886d393509725d5222f2","docker-archive:/var/tmp/gcr_io_****_****_0_0_90_2076038_623683417.tar"],"msg":"child process failure","time":"2023-10-06T09:37:10.828Z","v":0}

{"name":"kubernetes-monitor","hostname":"snyk-kubernetes-monitor-8fdcf4ccc-mh4ls","pid":7,"level":50,"error":{"message":"`skopeo copy --dest-compress-level 6 --src-cert-dir /srv/app/certs docker://gcr.io/****/****@sha256:62a442d3d2cf1d72758994e66767f2f6dbc8d2e4b5d9886d393509725d5222f2 docker-archive:/var/tmp/gcr_io_****_****_0_0_90_2076038_623683417.tar` failed with code 1","name":"ChildProcessError","stack":"ChildProcessError: `skopeo copy --dest-compress-level 6 --src-cert-dir /srv/app/certs docker://gcr.io/****/****@sha256:62a442d3d2cf1d72758994e66767f2f6dbc8d2e4b5d9886d393509725d5222f2 docker-archive:/var/tmp/gcr_io_****_****_0_0_90_2076038_623683417.tar` failed with code 1\n    at ChildProcess.<anonymous> (/srv/app/node_modules/child-process-promise/lib/index.js:132:23)\n    at ChildProcess.emit (node:events:513:28)\n    at ChildProcess.emit (node:domain:489:12)\n    at maybeClose (node:internal/child_process:1100:16)\n    at Process.ChildProcess._handle.onexit (node:internal/child_process:304:5)","code":1},"image":"gcr.io/****/****@sha256:62a442d3d2cf1d72758994e66767f2f6dbc8d2e4b5d9886d393509725d5222f2","msg":"failed to pull image docker/oci archive image","time":"2023-10-06T09:37:10.829Z","v":0}

Steps to reproduce

I have private GCP container registries, i created dockercfg.json which includes

  "credHelpers": {
    "us.gcr.io": "gcloud",
    "asia.gcr.io": "gcloud",
    "marketplace.gcr.io": "gcloud",
    "gcr.io": "gcloud",
    "eu.gcr.io": "gcloud",
    "staging-k8s.gcr.io": "gcloud"
  }

How a was able to fix this issue

Added to Deployment

  extraVolumes:
    - name: config-gcloud
      emptyDir:
         sizeLimit: 500Mi

  extraVolumeMounts:
    - name: config-gcloud
      mountPath: /srv/app/.config/gcloud

[johnjelinek]

I see from the deployment that this env var is set:

CLOUDSDK_CONFIG: /var/tmp/gcloud

And I even see logs in /var/tmp/gcloud/logs/. So, I wonder why node isn't putting logs in there. I bet CLOUDSDK_CONFIG needs to be passed here:

kubernetes-monitor/src/scanner/images/skopeo.ts

Lines 73 to 81 in 8e6f96c

	const env: Record<string, string | undefined> = {
	// The Azure CR credentials helper requires these env vars:
	AZURE_CLIENT_ID: process.env.AZURE_CLIENT_ID,
	AZURE_TENANT_ID: process.env.AZURE_TENANT_ID,
	AZURE_FEDERATED_TOKEN_FILE: process.env.AZURE_FEDERATED_TOKEN_FILE,
	AZURE_FEDERATED_TOKEN: process.env.AZURE_FEDERATED_TOKEN,
	AZURE_AUTHORITY_HOST: process.env.AZURE_AUTHORITY_HOST,
	};
	await processWrapper.exec('skopeo', env, ...args);

[mark-adams]

For clusters running in GKE, you can use GKE workload identity along with the normal cred helpers in dockercfg.json to authenticate with GCP registries:

{
  "credHelpers": {
    "gcr.io": "gcloud",
    "us.gcr.io": "gcloud",
    "eu.gcr.io": "gcloud",
    "asia.gcr.io": "gcloud",
    "staging-k8s.gcr.io": "gcloud",
    "marketplace.gcr.io": "gcloud"
  }
}

[johnjelinek]

That's the same config as in the OP. Is the volume mount override still required to get it to work with workload identity? It doesn't look like CLOUDSDK_CONFIG config var is supported yet.

[mark-adams]

I did not have to configure the CLOUDSDK_CONFIG variable but i did have to configure the additional volumes for /srv/app/.config/gcloud