Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

stat /argo/podmetadata/annotations: no such file or directory #5656

Closed
tommy-heyde-olsen opened this issue Apr 12, 2021 · 9 comments · Fixed by #5695
Closed

stat /argo/podmetadata/annotations: no such file or directory #5656

tommy-heyde-olsen opened this issue Apr 12, 2021 · 9 comments · Fixed by #5695
Assignees
@alexec
Labels
type/bug
Milestone
v3.0

Comments

@tommy-heyde-olsen
Copy link

tommy-heyde-olsen commented Apr 12, 2021 •
edited
Loading

Summary

After upgrade to v3.0.1 the wait container consistently fails the workflow with the following output:

time="2021-04-12T06:50:27.396Z" level=info msg="Starting Workflow Executor" version="{v3.0.1 2021-04-01T19:21:30Z a8c7d54c47b8dc08fd94d8347802d8d0604b09c3 v3.0.1 clean go1.15.7 gc linux/amd64}"
I0412 06:50:27.396709      11 merged_client_builder.go:121] Using in-cluster configuration
I0412 06:50:27.396871      11 merged_client_builder.go:163] Using in-cluster namespace
time="2021-04-12T06:50:27.399Z" level=info msg="Creating PNS executor (namespace: argo, pod: redacted-55vtt-2180130415, pid: 11)"
time="2021-04-12T06:50:27.399Z" level=info msg="Creating a K8sAPI executor"
time="2021-04-12T06:50:27.399Z" level=info msg="Executor (version: v3.0.1, build_date: 2021-04-01T19:21:30Z) initialized (pod: argo/redacted-55vtt-2180130415) with template:\n{\"name\":\"checkout\",\"inputs\":{\"parameters\":[{\"name\":\"cmd\",\"value\":\"rm -rf /src/.git/lfs\\n\"}]},\"outputs\":{\"parameters\":[{\"name\":\"components\",\"valueFrom\":{\"path\":\"/src/components.json\"}}],\"artifacts\":[{\"name\":\"source\",\"path\":\"/src\"}]},\"metadata\":{\"annotations\":{\"vault.security.banzaicloud.io/vault-role\":\"redacted\"}},\"container\":{\"name\":\"\",\"image\":\"redacted/ci-runner-git-docker-client:latest\",\"command\":[\"/bin/sh\",\"-ex\",\"-c\"],\"args\":[\"git-checkout\\nrm -rf /src/.git/lfs\\n\\n\"],\"env\":[{\"name\":\"BB_SSH_KEY\",\"valueFrom\":{\"secretKeyRef\":{\"name\":\"bitbucketssh\",\"key\":\"ssh-ci\"}}},{\"name\":\"REPOSITORY\",\"value\":\"git@github.com:redacted/redacted.git\"},{\"name\":\"BRANCH_NAME\",\"value\":\"redacted\"},{\"name\":\"COMMIT_SHA\",\"value\":\"9a595ea5c4bc861d108766d88b93a4c22e3cd1dc\"},{\"name\":\"GIT_REPO_DIR\",\"value\":\"/src\"},{\"name\":\"GIT_HTTP_HOST\",\"value\":\"vault:redacted#http-hostname\"},{\"name\":\"GIT_HTTP_HOST_AUTH_USERNAME\",\"value\":\"vault:redacted#http-auth-username\"},{\"name\":\"GIT_HTTP_HOST_AUTH_PASSWORD\",\"value\":\"vault:redacted#http-auth-password\"}],\"resources\":{\"limits\":{\"cpu\":\"1\"},\"requests\":{\"cpu\":\"1\"}}},\"archiveLocation\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio.argo.svc.cluster.local:9000\",\"bucket\":\"argo-artifacts\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"minio-azure-blob-secrets\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"minio-azure-blob-secrets\",\"key\":\"secretkey\"},\"key\":\"redacted-55vtt/redacted-55vtt-2180130415\"}},\"retryStrategy\":{\"limit\":3,\"retryPolicy\":\"Always\"}}"
time="2021-04-12T06:50:27.399Z" level=info msg="Starting annotations monitor"
time="2021-04-12T06:50:27.399Z" level=info msg="Starting deadline monitor"
time="2021-04-12T06:50:27.551Z" level=warning msg="failed to secure root file handle for 30" error="open /proc/30/root: permission denied"
time="2021-04-12T06:50:27.601Z" level=info msg="secured root for pid 30 root: vault-env"
time="2021-04-12T06:50:27.601Z" level=info msg="mapped container name \"main\" to pid 30"
time="2021-04-12T06:50:27.652Z" level=info msg="secured root for pid 30 root: vault-env"
time="2021-04-12T06:50:27.702Z" level=info msg="secured root for pid 30 root: vault-env"
time="2021-04-12T06:50:27.753Z" level=info msg="secured root for pid 30 root: vault-env"
time="2021-04-12T06:50:27.803Z" level=info msg="secured root for pid 30 root: vault-env"
time="2021-04-12T06:50:27.854Z" level=info msg="secured root for pid 30 root: vault-env"
time="2021-04-12T06:50:27.905Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:27.956Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:28.007Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:28.057Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:28.108Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:28.159Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:28.210Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:28.261Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:28.311Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:28.362Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:28.399Z" level=info msg="Waiting for \"main\" pid 30 to complete"
time="2021-04-12T06:50:28.413Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:28.464Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:28.515Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:28.566Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:28.617Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:28.667Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:28.719Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:28.771Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:28.821Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:28.872Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:28.923Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:28.973Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:29.024Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:29.075Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:29.127Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:29.178Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:29.229Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:29.279Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:29.330Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:29.380Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:29.431Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:29.482Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:29.532Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:29.583Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:29.633Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:29.685Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:29.735Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:29.786Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:29.837Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:29.888Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:29.939Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:29.991Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:30.042Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:30.093Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:30.144Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:30.197Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:30.247Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:30.298Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:30.349Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:30.399Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:30.450Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:30.501Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:30.551Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:30.602Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:30.652Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:30.703Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:30.753Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:30.804Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:30.855Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:30.906Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:30.957Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:31.007Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:31.058Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:31.109Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:31.159Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:31.210Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:31.260Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:31.311Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:31.362Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:31.412Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:31.463Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:31.514Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:31.564Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:31.615Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:31.666Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:31.716Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:31.767Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:31.818Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:31.868Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:31.919Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:31.970Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:32.021Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:32.072Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:32.122Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:32.173Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:32.223Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:32.274Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:32.325Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:32.375Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:32.426Z" level=info msg="secured root for pid 30 root: sh"
time="2021-04-12T06:50:46.401Z" level=info msg="\"main\" pid 30 completed"
time="2021-04-12T06:50:46.426Z" level=info msg="Watch pods 200"
I0412 06:50:46.426720      11 round_trippers.go:444] GET https://10.1.0.1:443/api/v1/namespaces/argo/pods?fieldSelector=metadata.name%3Dredacted-55vtt-2180130415&watch=true 200 OK in 25 milliseconds
time="2021-04-12T06:50:46.432Z" level=info msg="Main container completed"
time="2021-04-12T06:50:46.432Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2021-04-12T06:50:46.432Z" level=info msg="Capturing script exit code"
time="2021-04-12T06:50:46.432Z" level=info msg="Getting exit code of main"
time="2021-04-12T06:50:46.439Z" level=info msg="Get pods 200"
I0412 06:50:46.439467      11 round_trippers.go:444] GET https://10.1.0.1:443/api/v1/namespaces/argo/pods/redacted-55vtt-2180130415 200 OK in 6 milliseconds
time="2021-04-12T06:50:46.440Z" level=info msg="Saving logs"
time="2021-04-12T06:50:46.440Z" level=info msg="Getting output of main"
time="2021-04-12T06:50:46.452Z" level=info msg="List log 200"
I0412 06:50:46.452149      11 round_trippers.go:444] GET https://10.1.0.1:443/api/v1/namespaces/argo/pods/redacted-55vtt-2180130415/log?container=main&sinceTime= 200 OK in 11 milliseconds
time="2021-04-12T06:50:46.454Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: redacted-55vtt/redacted-55vtt-2180130415/main.log"
time="2021-04-12T06:50:46.454Z" level=info msg="Creating minio client minio.argo.svc.cluster.local:9000 using static credentials"
time="2021-04-12T06:50:46.454Z" level=info msg="Saving from /tmp/argo/outputs/logs/main.log to s3 (endpoint: minio.argo.svc.cluster.local:9000, bucket: argo-artifacts, key: redacted-55vtt/redacted-55vtt-2180130415/main.log)"
time="2021-04-12T06:50:46.660Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2021-04-12T06:50:46.660Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2021-04-12T06:50:46.660Z" level=info msg="Saving output parameters"
time="2021-04-12T06:50:46.660Z" level=info msg="Saving path output parameter: components"
time="2021-04-12T06:50:46.660Z" level=info msg="Copying /src/components.json from base image layer"
time="2021-04-12T06:50:46.660Z" level=info msg="Successfully saved output parameter: components"
time="2021-04-12T06:50:46.660Z" level=info msg="Saving output artifacts"
time="2021-04-12T06:50:46.661Z" level=info msg="Staging artifact: source"
time="2021-04-12T06:50:46.661Z" level=info msg="Copying /src from container base image layer to /tmp/argo/outputs/artifacts/source.tgz"
time="2021-04-12T06:50:46.661Z" level=info msg="Taring /src"
time="2021-04-12T06:50:46.661Z" level=debug msg="writing src"
time="2021-04-12T06:50:46.661Z" level=debug msg="writing src/.dockerignore"
time="2021-04-12T06:50:46.662Z" level=debug msg="writing src/.flake8"
... (more writes)
time="2021-04-12T06:50:47.399Z" level=fatal msg="stat /argo/podmetadata/annotations: no such file or directory"

This works perfectly fine in v2.12.11. Any idea why this is happening?

Diagnostics

What Kubernetes provider are you using?

Azure Kubernetes Service (AKS)

What version of Argo Workflows are you running?

v3.0.1

Paste the logs from the workflow controller:
There was nothing related to this in the workflow controller logs.

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.
@alexec
Copy link
Contributor

alexec commented Apr 12, 2021

Can you please attach the Pod's YAML?

@alexec alexec added this to the v3.0 milestone Apr 12, 2021
@terrytangyuan
Copy link
Member

I think I've seen this occasionally and it usually happens when the apiserver is unstable and fails to write/update the pod's annotations.

@tommy-heyde-olsen
Copy link
Author

tommy-heyde-olsen commented Apr 12, 2021 •
edited
Loading

Can you please attach the Pod's YAML?

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/limit-ranger: 'LimitRanger plugin set: memory request for container
      main; memory limit for container main'
    vault.security.banzaicloud.io/vault-role: redacted
    workflows.argoproj.io/execution: '{"deadline":"2021-04-12T07:48:57Z"}'
    workflows.argoproj.io/node-name: redacted-55vtt.checkout(3)
    workflows.argoproj.io/template: '{"name":"checkout","inputs":{"parameters":[{"name":"cmd","value":"rm
      -rf /src/.git/lfs\n"}]},"outputs":{"parameters":[{"name":"components","valueFrom":{"path":"/src/components.json"}}],"artifacts":[{"name":"source","path":"/src"}]},"metadata":{"annotations":{"vault.security.banzaicloud.io/vault-role":"redacted"}},"container":{"name":"","image":"redacted/ci-runner-git-docker-client:latest","command":["/bin/sh","-ex","-c"],"args":["git-checkout\nrm
      -rf /src/.git/lfs\n\n"],"env":[{"name":"BB_SSH_KEY","valueFrom":{"secretKeyRef":{"name":"bitbucketssh","key":"ssh-ci"}}},{"name":"REPOSITORY","value":"git@github.com:redacted.git"},{"name":"BRANCH_NAME","value":"release-global-tuning"},{"name":"COMMIT_SHA","value":"9a595ea5c4bc861d108766d88b93a4c22e3cd1dc"},{"name":"GIT_REPO_DIR","value":"/src"},{"name":"GIT_HTTP_HOST","value":"vault:redacted#http-hostname"},{"name":"GIT_HTTP_HOST_AUTH_USERNAME","value":"vault:redacted#http-auth-username"},{"name":"GIT_HTTP_HOST_AUTH_PASSWORD","value":"vault:redacted#http-auth-password"}],"resources":{"limits":{"cpu":"1"},"requests":{"cpu":"1"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio.argo.svc.cluster.local:9000","bucket":"argo-artifacts","insecure":true,"accessKeySecret":{"name":"minio-azure-blob-secrets","key":"accesskey"},"secretKeySecret":{"name":"minio-azure-blob-secrets","key":"secretkey"},"key":"redacted-55vtt/redacted-55vtt-2180130415"}},"retryStrategy":{"limit":3,"retryPolicy":"Always"}}'
  creationTimestamp: "2021-04-12T06:50:26Z"
  labels:
    product: forecasting
    workflows.argoproj.io/completed: "true"
    workflows.argoproj.io/workflow: redacted-55vtt
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:vault.security.banzaicloud.io/vault-role: {}
          f:workflows.argoproj.io/execution: {}
          f:workflows.argoproj.io/node-name: {}
          f:workflows.argoproj.io/template: {}
        f:labels:
          .: {}
          f:workflows.argoproj.io/completed: {}
          f:workflows.argoproj.io/workflow: {}
        f:ownerReferences:
          .: {}
          k:{"uid":"b22704b7-f3e2-4afd-b036-406104496195"}:
            .: {}
            f:apiVersion: {}
            f:blockOwnerDeletion: {}
            f:controller: {}
            f:kind: {}
            f:name: {}
            f:uid: {}
      f:spec:
        f:activeDeadlineSeconds: {}
        f:affinity:
          .: {}
          f:nodeAffinity:
            .: {}
            f:requiredDuringSchedulingIgnoredDuringExecution:
              .: {}
              f:nodeSelectorTerms: {}
        f:containers:
          k:{"name":"main"}:
            .: {}
            f:args: {}
            f:command: {}
            f:env:
              .: {}
              k:{"name":"ARGO_CONTAINER_NAME"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"BB_SSH_KEY"}:
                .: {}
                f:name: {}
                f:valueFrom:
                  .: {}
                  f:secretKeyRef:
                    .: {}
                    f:key: {}
                    f:name: {}
              k:{"name":"BRANCH_NAME"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"COMMIT_SHA"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"GIT_HTTP_HOST"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"GIT_HTTP_HOST_AUTH_PASSWORD"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"GIT_HTTP_HOST_AUTH_USERNAME"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"GIT_REPO_DIR"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"REPOSITORY"}:
                .: {}
                f:name: {}
                f:value: {}
            f:image: {}
            f:imagePullPolicy: {}
            f:name: {}
            f:resources:
              .: {}
              f:limits:
                .: {}
                f:cpu: {}
              f:requests:
                .: {}
                f:cpu: {}
            f:terminationMessagePath: {}
            f:terminationMessagePolicy: {}
          k:{"name":"wait"}:
            .: {}
            f:args: {}
            f:command: {}
            f:env:
              .: {}
              k:{"name":"ARGO_CONTAINER_NAME"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"ARGO_CONTAINER_RUNTIME_EXECUTOR"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"ARGO_POD_NAME"}:
                .: {}
                f:name: {}
                f:valueFrom:
                  .: {}
                  f:fieldRef:
                    .: {}
                    f:apiVersion: {}
                    f:fieldPath: {}
              k:{"name":"GODEBUG"}:
                .: {}
                f:name: {}
                f:value: {}
            f:image: {}
            f:imagePullPolicy: {}
            f:name: {}
            f:resources:
              .: {}
              f:limits:
                .: {}
                f:cpu: {}
                f:memory: {}
              f:requests:
                .: {}
                f:cpu: {}
                f:memory: {}
            f:securityContext:
              .: {}
              f:capabilities:
                .: {}
                f:add: {}
            f:terminationMessagePath: {}
            f:terminationMessagePolicy: {}
            f:volumeMounts:
              .: {}
              k:{"mountPath":"/argo/podmetadata"}:
                .: {}
                f:mountPath: {}
                f:name: {}
              k:{"mountPath":"/argo/secret/minio-azure-blob-secrets"}:
                .: {}
                f:mountPath: {}
                f:name: {}
                f:readOnly: {}
        f:dnsPolicy: {}
        f:enableServiceLinks: {}
        f:imagePullSecrets:
          .: {}
          k:{"name":"redacted"}:
            .: {}
            f:name: {}
        f:restartPolicy: {}
        f:schedulerName: {}
        f:securityContext: {}
        f:shareProcessNamespace: {}
        f:terminationGracePeriodSeconds: {}
        f:tolerations: {}
        f:volumes:
          .: {}
          k:{"name":"minio-azure-blob-secrets"}:
            .: {}
            f:name: {}
            f:secret:
              .: {}
              f:defaultMode: {}
              f:items: {}
              f:secretName: {}
          k:{"name":"podmetadata"}:
            .: {}
            f:downwardAPI:
              .: {}
              f:defaultMode: {}
              f:items: {}
            f:name: {}
    manager: workflow-controller
    operation: Update
    time: "2021-04-12T06:50:25Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
          k:{"type":"ContainersReady"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
          k:{"type":"Initialized"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
          k:{"type":"Ready"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
        f:containerStatuses: {}
        f:hostIP: {}
        f:initContainerStatuses: {}
        f:phase: {}
        f:podIP: {}
        f:podIPs:
          .: {}
          k:{"ip":"10.200.0.232"}:
            .: {}
            f:ip: {}
        f:startTime: {}
    manager: kubelet
    operation: Update
    time: "2021-04-12T06:50:46Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          f:product: {}
    manager: kubectl-label
    operation: Update
    time: "2021-04-12T07:00:30Z"
  name: redacted-55vtt-2180130415
  namespace: argo
  ownerReferences:
  - apiVersion: argoproj.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Workflow
    name: redacted-55vtt
    uid: b22704b7-f3e2-4afd-b036-406104496195
  resourceVersion: "105173111"
  selfLink: /api/v1/namespaces/argo/pods/redacted-55vtt-2180130415
  uid: d20467ec-133c-4d39-a9dd-56d3583ac68c
spec:
  activeDeadlineSeconds: 3511
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: agentpool
            operator: In
            values:
            - argo
            - argospot
            - batch
        - matchExpressions:
          - key: nodetype
            operator: In
            values:
            - argospot
  containers:
  - args:
    - --loglevel
    - debug
    - --gloglevel
    - "6"
    command:
    - argoexec
    - wait
    - --loglevel
    - info
    env:
    - name: ARGO_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: GODEBUG
      value: x509ignoreCN=0
    - name: ARGO_CONTAINER_RUNTIME_EXECUTOR
      value: pns
    - name: ARGO_CONTAINER_NAME
      value: wait
    image: argoproj/argoexec:v3.0.1
    imagePullPolicy: IfNotPresent
    name: wait
    resources:
      limits:
        cpu: 500m
        memory: 4Gi
      requests:
        cpu: 100m
        memory: 64Mi
    securityContext:
      capabilities:
        add:
        - SYS_PTRACE
        - SYS_CHROOT
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /argo/podmetadata
      name: podmetadata
    - mountPath: /argo/secret/minio-azure-blob-secrets
      name: minio-azure-blob-secrets
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-xmzwt
      readOnly: true
  - args:
    - /bin/sh
    - -ex
    - -c
    - |+
      git-checkout
      rm -rf /src/.git/lfs

    command:
    - /vault/vault-env
    env:
    - name: BB_SSH_KEY
      valueFrom:
        secretKeyRef:
          key: ssh-ci
          name: bitbucketssh
    - name: REPOSITORY
      value: git@github.com:redacted.git
    - name: BRANCH_NAME
      value: release-global-tuning
    - name: COMMIT_SHA
      value: 9a595ea5c4bc861d108766d88b93a4c22e3cd1dc
    - name: GIT_REPO_DIR
      value: /src
    - name: GIT_HTTP_HOST
      value: vault:redacted#http-hostname
    - name: GIT_HTTP_HOST_AUTH_USERNAME
      value: vault:redacted#http-auth-username
    - name: GIT_HTTP_HOST_AUTH_PASSWORD
      value: vault:redacted#http-auth-password
    - name: ARGO_CONTAINER_NAME
      value: main
    - name: VAULT_ADDR
      value: https://redacted
    - name: VAULT_SKIP_VERIFY
      value: "true"
    - name: VAULT_AUTH_METHOD
      value: jwt
    - name: VAULT_PATH
      value: redacted
    - name: VAULT_ROLE
      value: redacted
    - name: VAULT_IGNORE_MISSING_SECRETS
      value: "false"
    - name: VAULT_ENV_PASSTHROUGH
    - name: VAULT_JSON_LOG
      value: "false"
    - name: VAULT_CLIENT_TIMEOUT
      value: 30s
    - name: VAULT_LOG_LEVEL
      value: info
    image: redacted/ci-runner-git-docker-client:latest
    imagePullPolicy: Always
    name: main
    resources:
      limits:
        cpu: "1"
        memory: 500Mi
      requests:
        cpu: "1"
        memory: 500Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-xmzwt
      readOnly: true
    - mountPath: /vault/
      name: vault-env
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
  - name: redacted
  initContainers:
  - command:
    - sh
    - -c
    - cp /usr/local/bin/vault-env /vault/
    image: banzaicloud/vault-env:1.6.0
    imagePullPolicy: IfNotPresent
    name: copy-vault-env
    resources:
      limits:
        cpu: 250m
        memory: 64Mi
      requests:
        cpu: 50m
        memory: 64Mi
    securityContext:
      allowPrivilegeEscalation: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /vault/
      name: vault-env
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-xmzwt
      readOnly: true
  nodeName: aks-argospot-29846930-vmss00001a
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  shareProcessNamespace: true
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoSchedule
    key: argo
    operator: Exists
  - effect: NoSchedule
    key: kubernetes.azure.com/scalesetpriority
    operator: Equal
    value: spot
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  - effect: NoSchedule
    key: node.kubernetes.io/memory-pressure
    operator: Exists
  volumes:
  - downwardAPI:
      defaultMode: 420
      items:
      - fieldRef:
          apiVersion: v1
          fieldPath: metadata.annotations
        path: annotations
    name: podmetadata
  - name: minio-azure-blob-secrets
    secret:
      defaultMode: 420
      items:
      - key: accesskey
        path: accesskey
      - key: secretkey
        path: secretkey
      secretName: minio-azure-blob-secrets
  - name: default-token-xmzwt
    secret:
      defaultMode: 420
      secretName: default-token-xmzwt
  - emptyDir:
      medium: Memory
    name: vault-env
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-04-12T06:50:27Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2021-04-12T06:50:46Z"
    message: 'containers with unready status: [wait main]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2021-04-12T06:50:46Z"
    message: 'containers with unready status: [wait main]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2021-04-12T06:50:26Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://4020e9a87c551be179a85aba6c2427786a56b8010218f0dd90ee2e595c9fb9d5
    image: redacted/ci-runner-git-docker-client:latest
    imageID: redacted/ci-runner-git-docker-client@sha256:a85cc08af24b3cbb14a53c9131a97002c7ade63cf364e8e737c126c988be28f6
    lastState: {}
    name: main
    ready: false
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: containerd://4020e9a87c551be179a85aba6c2427786a56b8010218f0dd90ee2e595c9fb9d5
        exitCode: 0
        finishedAt: "2021-04-12T06:50:45Z"
        reason: Completed
        startedAt: "2021-04-12T06:50:27Z"
  - containerID: containerd://26602a850a7b1e7e5732b4be83af79b9f4360c62b4f74c90dfa632b5c08348b2
    image: docker.io/argoproj/argoexec:v3.0.1
    imageID: docker.io/argoproj/argoexec@sha256:e96900bb9f6c8d2a3b1c74bcb93117187dafa2291446d20f590056aed878be67
    lastState: {}
    name: wait
    ready: false
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: containerd://26602a850a7b1e7e5732b4be83af79b9f4360c62b4f74c90dfa632b5c08348b2
        exitCode: 1
        finishedAt: "2021-04-12T06:50:47Z"
        reason: Error
        startedAt: "2021-04-12T06:50:27Z"
  hostIP: 10.200.0.155
  initContainerStatuses:
  - containerID: containerd://b5a5750ef6477339d6ddc3e17a690a71cad37308c097ad7f6c549b9dc9a7ce00
    image: docker.io/banzaicloud/vault-env:1.6.0
    imageID: docker.io/banzaicloud/vault-env@sha256:c6b3d04a0cfc8648b4a67f034dc70d549f4dbd6b7d5c82584c3fa8324537583d
    lastState: {}
    name: copy-vault-env
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: containerd://b5a5750ef6477339d6ddc3e17a690a71cad37308c097ad7f6c549b9dc9a7ce00
        exitCode: 0
        finishedAt: "2021-04-12T06:50:27Z"
        reason: Completed
        startedAt: "2021-04-12T06:50:26Z"
  phase: Failed
  podIP: 10.200.0.232
  podIPs:
  - ip: 10.200.0.232
  qosClass: Burstable
  startTime: "2021-04-12T06:50:26Z"

@tommy-heyde-olsen
Copy link
Author

tommy-heyde-olsen commented Apr 12, 2021 •
edited
Loading

I think I've seen this occasionally and it usually happens when the apiserver is unstable and fails to write/update the pod's annotations.

This is a very interesting theory; we have certainly seen this behaviour before with AKS.

However we have consistently seen that one workflow fails with this error 100% of the times we've run it, but another workflow with a similar step did not fail. Which contradicts that it is the apiserver which is faulty, because then it would give this error on only some of the runs and not consistently on one workflow.

@alexec
Copy link
Contributor

alexec commented Apr 12, 2021

Can you please try in v2.12?

@terrytangyuan
Copy link
Member

terrytangyuan commented Apr 12, 2021 •
edited
Loading

Okay then the causes might be different. In our case, we only observed this when apiserver was under extremely high load or unstable. We observed this in v2.12.

@tommy-heyde-olsen
Copy link
Author

tommy-heyde-olsen commented Apr 12, 2021 •
edited
Loading

Can you please try in v2.12?

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/limit-ranger: 'LimitRanger plugin set: memory request for container
      main; memory limit for container main'
    vault.security.banzaicloud.io/vault-role: redacted
    workflows.argoproj.io/execution: '{"deadline":"2021-04-12T16:17:59Z"}'
    workflows.argoproj.io/node-name: redacted-g8sfx.checkout(0)
    workflows.argoproj.io/template: '{"name":"checkout","arguments":{},"inputs":{"parameters":[{"name":"cmd","value":"rm
      -rf /src/.git/lfs\n"}]},"outputs":{"parameters":[{"name":"components","valueFrom":{"path":"/src/components.json"}}],"artifacts":[{"name":"source","path":"/src"}]},"metadata":{"annotations":{"vault.security.banzaicloud.io/vault-role":"redacted"}},"container":{"name":"","image":"redacted/ci-runner-git-docker-client:latest","command":["/bin/sh","-ex","-c"],"args":["git-checkout\nrm
      -rf /src/.git/lfs\n\n"],"env":[{"name":"BB_SSH_KEY","valueFrom":{"secretKeyRef":{"name":"bitbucketssh","key":"ssh-ci"}}},{"name":"REPOSITORY","value":"git@github.com:redacted.git"},{"name":"BRANCH_NAME","value":"release-global-tuning"},{"name":"COMMIT_SHA","value":"9a595ea5c4bc861d108766d88b93a4c22e3cd1dc"},{"name":"GIT_REPO_DIR","value":"/src"},{"name":"GIT_HTTP_HOST","value":"vault:redacted#http-hostname"},{"name":"GIT_HTTP_HOST_AUTH_USERNAME","value":"vault:redacted#http-auth-username"},{"name":"GIT_HTTP_HOST_AUTH_PASSWORD","value":"vault:redacted#http-auth-password"}],"resources":{"limits":{"cpu":"1"},"requests":{"cpu":"1"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio.argo.svc.cluster.local:9000","bucket":"argo-artifacts","insecure":true,"accessKeySecret":{"name":"minio-azure-blob-secrets","key":"accesskey"},"secretKeySecret":{"name":"minio-azure-blob-secrets","key":"secretkey"},"key":"redacted-g8sfx/redacted-g8sfx-1027290098"}},"retryStrategy":{"limit":3,"retryPolicy":"Always"}}'
  creationTimestamp: "2021-04-12T15:18:00Z"
  labels:
    workflows.argoproj.io/completed: "false"
    workflows.argoproj.io/workflow: redacted-g8sfx
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:vault.security.banzaicloud.io/vault-role: {}
          f:workflows.argoproj.io/execution: {}
          f:workflows.argoproj.io/node-name: {}
          f:workflows.argoproj.io/template: {}
        f:labels:
          .: {}
          f:workflows.argoproj.io/completed: {}
          f:workflows.argoproj.io/workflow: {}
        f:ownerReferences:
          .: {}
          k:{"uid":"dd8c7b22-0cd8-474e-9e4b-e51750550803"}:
            .: {}
            f:apiVersion: {}
            f:blockOwnerDeletion: {}
            f:controller: {}
            f:kind: {}
            f:name: {}
            f:uid: {}
      f:spec:
        f:activeDeadlineSeconds: {}
        f:affinity:
          .: {}
          f:nodeAffinity:
            .: {}
            f:requiredDuringSchedulingIgnoredDuringExecution:
              .: {}
              f:nodeSelectorTerms: {}
        f:containers:
          k:{"name":"main"}:
            .: {}
            f:args: {}
            f:command: {}
            f:env:
              .: {}
              k:{"name":"BB_SSH_KEY"}:
                .: {}
                f:name: {}
                f:valueFrom:
                  .: {}
                  f:secretKeyRef:
                    .: {}
                    f:key: {}
                    f:name: {}
              k:{"name":"BRANCH_NAME"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"COMMIT_SHA"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"GIT_HTTP_HOST"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"GIT_HTTP_HOST_AUTH_PASSWORD"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"GIT_HTTP_HOST_AUTH_USERNAME"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"GIT_REPO_DIR"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"REPOSITORY"}:
                .: {}
                f:name: {}
                f:value: {}
            f:image: {}
            f:imagePullPolicy: {}
            f:name: {}
            f:resources:
              .: {}
              f:limits:
                .: {}
                f:cpu: {}
              f:requests:
                .: {}
                f:cpu: {}
            f:terminationMessagePath: {}
            f:terminationMessagePolicy: {}
          k:{"name":"wait"}:
            .: {}
            f:args: {}
            f:command: {}
            f:env:
              .: {}
              k:{"name":"ARGO_CONTAINER_RUNTIME_EXECUTOR"}:
                .: {}
                f:name: {}
                f:value: {}
              k:{"name":"ARGO_POD_NAME"}:
                .: {}
                f:name: {}
                f:valueFrom:
                  .: {}
                  f:fieldRef:
                    .: {}
                    f:apiVersion: {}
                    f:fieldPath: {}
              k:{"name":"GODEBUG"}:
                .: {}
                f:name: {}
                f:value: {}
            f:image: {}
            f:imagePullPolicy: {}
            f:name: {}
            f:resources:
              .: {}
              f:limits:
                .: {}
                f:cpu: {}
                f:memory: {}
              f:requests:
                .: {}
                f:cpu: {}
                f:memory: {}
            f:securityContext:
              .: {}
              f:capabilities:
                .: {}
                f:add: {}
            f:terminationMessagePath: {}
            f:terminationMessagePolicy: {}
            f:volumeMounts:
              .: {}
              k:{"mountPath":"/argo/podmetadata"}:
                .: {}
                f:mountPath: {}
                f:name: {}
              k:{"mountPath":"/argo/secret/minio-azure-blob-secrets"}:
                .: {}
                f:mountPath: {}
                f:name: {}
                f:readOnly: {}
        f:dnsPolicy: {}
        f:enableServiceLinks: {}
        f:imagePullSecrets:
          .: {}
          k:{"name":"redacted"}:
            .: {}
            f:name: {}
        f:restartPolicy: {}
        f:schedulerName: {}
        f:securityContext: {}
        f:shareProcessNamespace: {}
        f:terminationGracePeriodSeconds: {}
        f:tolerations: {}
        f:volumes:
          .: {}
          k:{"name":"minio-azure-blob-secrets"}:
            .: {}
            f:name: {}
            f:secret:
              .: {}
              f:defaultMode: {}
              f:items: {}
              f:secretName: {}
          k:{"name":"podmetadata"}:
            .: {}
            f:downwardAPI:
              .: {}
              f:defaultMode: {}
              f:items: {}
            f:name: {}
    manager: workflow-controller
    operation: Update
    time: "2021-04-12T15:17:59Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
          k:{"type":"ContainersReady"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
          k:{"type":"Initialized"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
          k:{"type":"Ready"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
        f:containerStatuses: {}
        f:hostIP: {}
        f:initContainerStatuses: {}
        f:phase: {}
        f:podIP: {}
        f:podIPs:
          .: {}
          k:{"ip":"10.200.0.186"}:
            .: {}
            f:ip: {}
        f:startTime: {}
    manager: kubelet
    operation: Update
    time: "2021-04-12T15:18:03Z"
  name: redacted-g8sfx-1027290098
  namespace: argo
  ownerReferences:
  - apiVersion: argoproj.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Workflow
    name: redacted-g8sfx
    uid: dd8c7b22-0cd8-474e-9e4b-e51750550803
  resourceVersion: "105416259"
  selfLink: /api/v1/namespaces/argo/pods/redacted-g8sfx-1027290098
  uid: d4a7d092-11a3-473a-90ae-c0b7ad5d91dc
spec:
  activeDeadlineSeconds: 3599
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: agentpool
            operator: In
            values:
            - argo
            - argospot
            - batch
        - matchExpressions:
          - key: nodetype
            operator: In
            values:
            - argospot
  containers:
  - args:
    - --loglevel
    - debug
    - --gloglevel
    - "6"
    command:
    - argoexec
    - wait
    env:
    - name: ARGO_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: GODEBUG
      value: x509ignoreCN=0
    - name: ARGO_CONTAINER_RUNTIME_EXECUTOR
      value: pns
    image: argoproj/argoexec:v2.12.11
    imagePullPolicy: IfNotPresent
    name: wait
    resources:
      limits:
        cpu: 500m
        memory: 4Gi
      requests:
        cpu: 100m
        memory: 64Mi
    securityContext:
      capabilities:
        add:
        - SYS_PTRACE
        - SYS_CHROOT
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /argo/podmetadata
      name: podmetadata
    - mountPath: /argo/secret/minio-azure-blob-secrets
      name: minio-azure-blob-secrets
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-xmzwt
      readOnly: true
  - args:
    - /bin/sh
    - -ex
    - -c
    - |+
      git-checkout
      rm -rf /src/.git/lfs

    command:
    - /vault/vault-env
    env:
    - name: BB_SSH_KEY
      valueFrom:
        secretKeyRef:
          key: ssh-ci
          name: bitbucketssh
    - name: REPOSITORY
      value: git@redacted/redacted.git
    - name: BRANCH_NAME
      value: release-global-tuning
    - name: COMMIT_SHA
      value: 9a595ea5c4bc861d108766d88b93a4c22e3cd1dc
    - name: GIT_REPO_DIR
      value: /src
    - name: GIT_HTTP_HOST
      value: vault:redacted#http-hostname
    - name: GIT_HTTP_HOST_AUTH_USERNAME
      value: vault:redacted#http-auth-username
    - name: GIT_HTTP_HOST_AUTH_PASSWORD
      value: vault:redacted#http-auth-password
    - name: VAULT_ADDR
      value: https://redacted
    - name: VAULT_SKIP_VERIFY
      value: "true"
    - name: VAULT_AUTH_METHOD
      value: jwt
    - name: VAULT_PATH
      value: redacted
    - name: VAULT_ROLE
      value: redacted
    - name: VAULT_IGNORE_MISSING_SECRETS
      value: "false"
    - name: VAULT_ENV_PASSTHROUGH
    - name: VAULT_JSON_LOG
      value: "false"
    - name: VAULT_CLIENT_TIMEOUT
      value: 30s
    - name: VAULT_LOG_LEVEL
      value: info
    image: redacted/ci-runner-git-docker-client:latest
    imagePullPolicy: Always
    name: main
    resources:
      limits:
        cpu: "1"
        memory: 500Mi
      requests:
        cpu: "1"
        memory: 500Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-xmzwt
      readOnly: true
    - mountPath: /vault/
      name: vault-env
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
  - name: redacted
  initContainers:
  - command:
    - sh
    - -c
    - cp /usr/local/bin/vault-env /vault/
    image: banzaicloud/vault-env:1.6.0
    imagePullPolicy: IfNotPresent
    name: copy-vault-env
    resources:
      limits:
        cpu: 250m
        memory: 64Mi
      requests:
        cpu: 50m
        memory: 64Mi
    securityContext:
      allowPrivilegeEscalation: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /vault/
      name: vault-env
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-xmzwt
      readOnly: true
  nodeName: aks-argospot-29846930-vmss00001a
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  shareProcessNamespace: true
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoSchedule
    key: argo
    operator: Exists
  - effect: NoSchedule
    key: kubernetes.azure.com/scalesetpriority
    operator: Equal
    value: spot
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  - effect: NoSchedule
    key: node.kubernetes.io/memory-pressure
    operator: Exists
  volumes:
  - downwardAPI:
      defaultMode: 420
      items:
      - fieldRef:
          apiVersion: v1
          fieldPath: metadata.annotations
        path: annotations
    name: podmetadata
  - name: minio-azure-blob-secrets
    secret:
      defaultMode: 420
      items:
      - key: accesskey
        path: accesskey
      - key: secretkey
        path: secretkey
      secretName: minio-azure-blob-secrets
  - name: default-token-xmzwt
    secret:
      defaultMode: 420
      secretName: default-token-xmzwt
  - emptyDir:
      medium: Memory
    name: vault-env
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-04-12T15:18:02Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2021-04-12T15:18:03Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2021-04-12T15:18:03Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2021-04-12T15:18:00Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://c63274f3abb84d5372aa69792af033e0d830b7199f4ba8c45f790a14bf662615
    image: redacted/ci-runner-git-docker-client:latest
    imageID: redacted/ci-runner-git-docker-client@sha256:a85cc08af24b3cbb14a53c9131a97002c7ade63cf364e8e737c126c988be28f6
    lastState: {}
    name: main
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2021-04-12T15:18:02Z"
  - containerID: containerd://b91ce4c7fcb2174c14aad5e6f64cfb4dbb39fecbb3de0415eff6588d51037549
    image: docker.io/argoproj/argoexec:v2.12.11
    imageID: docker.io/argoproj/argoexec@sha256:38409ff2bb27b15c4f3184d436c7893062ae2df146f46846da92919883fd7b3b
    lastState: {}
    name: wait
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2021-04-12T15:18:02Z"
  hostIP: 10.200.0.155
  initContainerStatuses:
  - containerID: containerd://513a768d7748e02897e167d78cd0287a19424fc66f9c928379e8a5260a97aa30
    image: docker.io/banzaicloud/vault-env:1.6.0
    imageID: docker.io/banzaicloud/vault-env@sha256:c6b3d04a0cfc8648b4a67f034dc70d549f4dbd6b7d5c82584c3fa8324537583d
    lastState: {}
    name: copy-vault-env
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: containerd://513a768d7748e02897e167d78cd0287a19424fc66f9c928379e8a5260a97aa30
        exitCode: 0
        finishedAt: "2021-04-12T15:18:01Z"
        reason: Completed
        startedAt: "2021-04-12T15:18:01Z"
  phase: Running
  podIP: 10.200.0.186
  podIPs:
  - ip: 10.200.0.186
  qosClass: Burstable
  startTime: "2021-04-12T15:18:00Z"

@alexec alexec removed this from the v3.0 milestone Apr 17, 2021
@alexec alexec added this to the v3.0 milestone Apr 17, 2021
@alexec alexec self-assigned this Apr 17, 2021
@alexec
Copy link
Contributor

alexec commented Apr 17, 2021

This might be a pre-existing timing issue surfaced introduced by changes in v3.

alexec added a commit to alexec/argo-workflows that referenced this issue Apr 17, 2021
@alexec
fix(executor): Ignore not existing metadata. Fixes argoproj#5656
3194ec1 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
@alexec alexec linked a pull request Apr 17, 2021 that will close this issue
fix(executor): Ignore not existing metadata. Fixes #5656 #5695
Merged
@tommy-heyde-olsen
Copy link
Author

Thanks for taking the time to look into this.

alexec added a commit that referenced this issue Apr 19, 2021
@alexec
fix(executor): Ignore not existing metadata. Fixes #5656 (#5695)
9a872de
@simster7 simster7 mentioned this issue Apr 19, 2021
Closed
50 tasks
simster7 pushed a commit that referenced this issue Apr 19, 2021
@alexec @simster7
fix(executor): Ignore not existing metadata. Fixes #5656 (#5695)
f720040
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alexec alexec

Labels
type/bug
Projects
None yet
Milestone
v3.0
Development

Successfully merging a pull request may close this issue.

fix(executor): Ignore not existing metadata. Fixes #5656 alexec/argo-workflows
3 participants
@alexec @terrytangyuan @tommy-heyde-olsen