UI could help in troubleshooting any pods stuck in ContainerCreating state

[mattnworb]

What happened:

If a pipeline is run where the pod for one of the steps is stuck in ContainerCreating or any other non-running state, the Pipeline UI is capable of showing what state the pod is in, but not why or what the user running the experiment should do to resolve things.

Additionally, an error message is shown about not being able to view the logs, which happens since there are no logs to display for a pod that has not yet run - but the phrasing is somewhat confusing.

What did you expect to happen:

It would be really helpful to ML engineers running an experiment for the UI to attempt to diagnose the problem, or at least to display the events from the Kubernetes pod - which in the case of the screenshot above, would show that it was due to the pod attempting to mount a Secret that does not actually exist.

As-is, the user running the experiment has to inspect the state of the Kubernetes cluster to troubleshoot the problem, and we have found that often the engineer running the experiment does not have the experience or background to be able to do so effectively.

Anything else you would like to add:
For what it is worth, this is less of a bug than feedback and a feature request.

[paveldournov]

@mattnworb do the stackdriver logs provide any info if you navigate to it using the links in the UI?

[mattnworb]

In the case where a pod can't be created because its PodSpec tries to mount a volume/secret that doesn't exist, I don't think the Stackdriver Logs help - because there are no logs from the container to show if none of the containers have run yet.

[mattnworb]

to add more context to the above comment, when a pod is in ContainerCreating, the UI shows two links:

Logs can still be viewed in either [Legacy Stackdriver] or in [Stackdriver Kubernetes Monitoring]

the former is to a Stackdriver Logging query like

resource.type="container"
resource.labels.cluster_name:...
resource.labels.pod_id:...

the latter is a query for

resource.type="k8s_container"
resource.labels.cluster_name:...
resource.labels.pod_name:...

since the container has not run, there are no logs from the container.

However an additional link to view the Kubernetes Events for the pod could be helpful:

resource.type="k8s_pod"
resource.labels.cluster_name=...
resource.labels.pod_name=...

since in the case of a volume not being able to be mounted because the referenced Secret doesn't exist, etc., an Event is generated and attached to the Pod.

[Bobgy]

I think this would be very useful for debugging, also aligns with #3112