Increase visibility to resources stuck in a terminating state

[GuessWhoSamFoo]

Describe the problem/challenge you have
xref: kubernetes/kubernetes#60807

Cases where resources can be stuck in a terminating state:

• an object being deleted has a finalizer
• a namespace being deleted contains a finalizer
• a namespace being part of an APIService (example with cert-manager deleting namespace stuck at "Terminating" state kubernetes/kubernetes#60807 (comment))

Describe the solution you'd like
Users of Kubernetes who are unaware of this issue would have to find the relevant Github issue to fix this. Octant can provide value by suggesting cleanup of orphaned apiservices or an interface to patch over an unused finalizer.

Environment:

• Octant version (use octant version): 0.16
• Kubernetes version (use kubectl version): 1.18.2
• OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): ubuntu

[scothis]

How do you distinguish "stuck" resources vs resources that are gracefully cleaning up external state before being fully deleted?

[GuessWhoSamFoo]

For namespaces specifically, we can highlight the condition (kubernetes/kubernetes#82189).

Octant doesn't have to distinguish the graceful case. It can show a warning to the effect of "if this resources is not terminating like you'd expect, consider looking at relevant finalizer/apiserver/controller". Maybe these is a way to be more specific

[wwitzel3]

Example of a terminating namespace status

status:
  conditions:
  - lastTransitionTime: "2021-07-14T18:19:44Z"
    message: 'Discovery failed for some groups, 1 failing: unable to retrieve the
      complete list of server APIs: admission.certmanager.k8s.io/v1beta1: the server
      is currently unable to handle the request'
    reason: DiscoveryFailed
    status: "True"
    type: NamespaceDeletionDiscoveryFailure
  - lastTransitionTime: "2021-07-14T18:19:44Z"
    message: All legacy kube types successfully parsed
    reason: ParsedGroupVersions
    status: "False"
    type: NamespaceDeletionGroupVersionParsingFailure
  - lastTransitionTime: "2021-07-14T18:19:44Z"
    message: All content successfully deleted, may be waiting on finalization
    reason: ContentDeleted
    status: "False"
    type: NamespaceDeletionContentFailure
  - lastTransitionTime: "2021-07-14T18:19:44Z"
    message: All content successfully removed
    reason: ContentRemoved
    status: "False"
    type: NamespaceContentRemaining
  - lastTransitionTime: "2021-07-14T18:19:44Z"
    message: All content-preserving finalizers finished
    reason: ContentHasNoFinalizers
    status: "False"
    type: NamespaceFinalizersRemaining
  phase: Terminating

[wwitzel3]

I think is less about guess if something is "stuck" and more about surfacing the conditions in a way that can help a user decide if they want to wait or intervene.

An idea that was discussed was looking at the lastTransitionTime of objects in the Terminating phase and using some delta of that to provider an extra layer of condition introspection.

[ftovaro]

What about having a toast that informs that a resource is stuck, it could have the link to the resource and with that you don't need to keep checking a table to know the state of the resource, instead, you get an alert that something is not right

[wwitzel3]

To test this behavior, adding a finalizer that you know will never finish.

• time threshold for terminating: 5 minutes (configurable via preferences)
• at threshold - fetch further details
• should part of existing object printer as a conditional
• UI suggestions
  • Object Summary
    • Add clear callout to the existing conditions table when threshold is met
  • Object List
    • Phase field has a an asterisks "Terminating *" and on hover you get a flyout?
    • Add a column that has conditions?

[lenriquez]

This may help for testing https://www.openshift.com/blog/the-hidden-dangers-of-terminating-namespaces