Google Secret Manager: Retry total timeout exceeded before any response was received

[saranicole]

External secrets are failing to sync with the following error:

{"level":50,"time":1589573368279,"pid":17,"hostname":"kubernetes-external-secrets-d79dc6dc5-4nhfq","code":4,"stack":"Error: Re │
│ try total timeout exceeded before any response was received\n    at repeat (/app/node_modules/google-gax/build/src/normalCalls │
│ /retries.js:65:31)\n    at Timeout._onTimeout (/app/node_modules/google-gax/build/src/normalCalls/retries.js:100:25)\n    at l │
│ istOnTimeout (internal/timers.js:549:17)\n    at processTimers (internal/timers.js:492:7)","type":"Error","msg":"failure while │
│  polling the secret kubernetes-external-secrets/ccl-mongo"}

Provider: Google Secret Manager
Kubernetes 1.15

ccl-mongo looks like this

---
apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: ccl-mongo
  namespace: kubernetes-external-secrets
spec:
  backendType: gcpSecretsManager
  projectId: security-tools
  data:
    - key: staging_rd_ccl_mongo
      name: mongo.txt
      version: latest

Any ideas on how to debug this issue?

[saranicole]

It seems that this is an issue at the google-gax level due to an API call failure

There is an open issue requesting better debug information for this very error
googleapis/gax-nodejs#741

Still not sure what is causing it though

[saranicole]

Turns out my cluster's kube-dns is broken - nslookup google.com fails. The problem is upstream of this tool. Closing.