Unable to attach or mount volumes: unmounted volumes=[mlpipeline-minio-artifact], unattached volumes=[podmetadata docker-sock mlpipeline-minio-artifact default-editor-token-8lmfr]: timed out waiting for the condition

[631068264]

kubectl version

[root@iZ1bb01rvtheuakv3h25ntZ ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.5+k3s2", GitCommit:"746cf4031370f443bf1230272bc79f2f72de2869", GitTreeState:"clean", BuildDate:"2020-12-18T01:41:55Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.5+k3s2", GitCommit:"746cf4031370f443bf1230272bc79f2f72de2869", GitTreeState:"clean", BuildDate:"2020-12-18T01:41:55Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

Pipelines 1.7.0 install kubeflow 1.4.1

pipeline keep pending execution

Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    28m                  default-scheduler  Successfully assigned kubeflow-user-example-com/my-test-pipeline-beta-bzrlr-3669637779 to iz1bb01rvtheuakv3h25ntz
  Warning  FailedMount  14m (x2 over 21m)    kubelet            Unable to attach or mount volumes: unmounted volumes=[mlpipeline-minio-artifact], unattached volumes=[kfp-launcher default-editor-token-8lmfr podmetadata docker-sock mlpipeline-minio-artifact]: timed out waiting for the condition
  Warning  FailedMount  12m (x2 over 23m)    kubelet            Unable to attach or mount volumes: unmounted volumes=[mlpipeline-minio-artifact], unattached volumes=[docker-sock mlpipeline-minio-artifact kfp-launcher default-editor-token-8lmfr podmetadata]: timed out waiting for the condition
  Warning  FailedMount  10m (x2 over 19m)    kubelet            Unable to attach or mount volumes: unmounted volumes=[mlpipeline-minio-artifact], unattached volumes=[mlpipeline-minio-artifact kfp-launcher default-editor-token-8lmfr podmetadata docker-sock]: timed out waiting for the condition
  Warning  FailedMount  8m1s                 kubelet            Unable to attach or mount volumes: unmounted volumes=[mlpipeline-minio-artifact], unattached volumes=[default-editor-token-8lmfr podmetadata docker-sock mlpipeline-minio-artifact kfp-launcher]: timed out waiting for the condition
  Warning  FailedMount  5m43s (x3 over 26m)  kubelet            Unable to attach or mount volumes: unmounted volumes=[mlpipeline-minio-artifact], unattached volumes=[podmetadata docker-sock mlpipeline-minio-artifact kfp-launcher default-editor-token-8lmfr]: timed out waiting for the condition
  Warning  FailedMount  96s (x21 over 28m)   kubelet            MountVolume.SetUp failed for volume "mlpipeline-minio-artifact" : secret "mlpipeline-minio-artifact" not found

Events:
  Type     Reason       Age                   From               Message
  ----     ------       ----                  ----               -------
  Normal   Scheduled    53m                   default-scheduler  Successfully assigned kubeflow-user-example-com/parameterized-tfx-oss-t48jj-3672522880 to 10-19-64-204
  Warning  FailedMount  40m (x4 over 49m)     kubelet            Unable to attach or mount volumes: unmounted volumes=[mlpipeline-minio-artifact], unattached volumes=[docker-sock mlpipeline-minio-artifact default-editor-token-8lmfr podmetadata]: timed out waiting for the condition
  Warning  FailedMount  36m (x2 over 38m)     kubelet            Unable to attach or mount volumes: unmounted volumes=[mlpipeline-minio-artifact], unattached volumes=[default-editor-token-8lmfr podmetadata docker-sock mlpipeline-minio-artifact]: timed out waiting for the condition
  Warning  FailedMount  33m (x3 over 51m)     kubelet            Unable to attach or mount volumes: unmounted volumes=[mlpipeline-minio-artifact], unattached volumes=[mlpipeline-minio-artifact default-editor-token-8lmfr podmetadata docker-sock]: timed out waiting for the condition
  Warning  FailedMount  8m46s (x5 over 31m)   kubelet            Unable to attach or mount volumes: unmounted volumes=[mlpipeline-minio-artifact], unattached volumes=[podmetadata docker-sock mlpipeline-minio-artifact default-editor-token-8lmfr]: timed out waiting for the condition
  Warning  FailedMount  2m59s (x33 over 53m)  kubelet            MountVolume.SetUp failed for volume "mlpipeline-minio-artifact" : secret "mlpipeline-minio-artifact" not found

[631068264]

finally I find that the pipeline job is in the A namespace and the mlpipeline-minio-artifact is in kubeflow namespace.I don't know why.

[zijianjoy]

Possibility related #4649.

What is the Kubernetes version you are running on, and what is the executor you are using? #5714

[631068264]

Kubernetes version v1.19.5+k3s2
I don't know what is the executor. Do you mean this gcr.io/ml-pipeline/kfp-launcher:1.7.1 . I find it in offical Tutorial V2 lightweight Python components

Finally I use pipeline sdk v1 and gcr.io/ml-pipeline/kfp-launcher:1.8.7 to run my pipeline task and it work well

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[andrejlukic]

I am experiencing the same issue. It looks random. I can actually write to the volume but then I do a minor code change (e.g. adding one meaningless line) and suddenly the pipeline gets stuck in pending state and the kubectl describe pod reports the same error as OP noted.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.