Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added proposal for remediation history #567

Closed
wants to merge 8 commits into from

Conversation

slintes
Copy link
Member

@slintes slintes commented Dec 15, 2020

We propose to track a limited history of remediations and showing it in the UI in order to allow admins to detect recurring node problems easier.

/cc @beekhof @JoelSpeed @michaelgugino @enxebre

Copy link
Member Author

@slintes slintes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jewzaam @JoelSpeed Thanks for your review! Please check my comments / improvements.

@openshift-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign spadgett after the PR has been reviewed.
You can assign the PR to them by writing /assign @spadgett in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@slintes slintes force-pushed the remediation-history branch from 3f6865a to 01afb4d Compare February 17, 2021 18:35
@slintes
Copy link
Member Author

slintes commented Feb 17, 2021

Before we go on discussing names of timestamps on outdated code snippets... is this fine for everyone?

	// the time when the unhealthy condition was detected
	Detected *metav1.Time `json:"detected"`

	// the time when remediation started, will be around "detected" plus the "timeout" configured on the MHC
	Started *metav1.Time `json:"started,omitempty"`

	// the time when the node is remediated and in a safe state, so that workloads can be rescheduled.
	Remediated *metav1.Time `json:"remediated,omitempty"`

Signed-off-by: Marc Sluiter <msluiter@redhat.com>
Signed-off-by: Marc Sluiter <msluiter@redhat.com>
Signed-off-by: Marc Sluiter <msluiter@redhat.com>
Signed-off-by: Marc Sluiter <msluiter@redhat.com>
Signed-off-by: Marc Sluiter <msluiter@redhat.com>
@slintes slintes force-pushed the remediation-history branch from 01afb4d to 44322ae Compare February 17, 2021 18:41
Signed-off-by: Marc Sluiter <msluiter@redhat.com>
@beekhof
Copy link
Contributor

beekhof commented Feb 18, 2021

@JoelSpeed anything holding this back from being merged?

Copy link
Contributor

@JoelSpeed JoelSpeed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I understand it, a machine will get detected, the remediation event will be added, that event will then be updated as time progresses. What happens if that remediation event never actually causes a remediation, does the status then look like we lost track of it?

Do we need to have a third Type that is called recovered and then if we detect a machine that we thought would go unhealthy, mark the event as recovered?
(If we go down this avenue, how does this work when nodes are starting, since they appear unhealthy at first?)

Has there been any discussion with the upstream community about this type of enhancement, would be good to have this feature upstreamed as well

…e field

Signed-off-by: Marc Sluiter <msluiter@redhat.com>
@slintes
Copy link
Member Author

slintes commented Feb 22, 2021

Renamed Remediation to RemediationEvent and further clarified values of the Type field (machineDeletion or external or aborted now)

Signed-off-by: Marc Sluiter <msluiter@redhat.com>
@openshift-bot
Copy link

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

@openshift-ci openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 13, 2021
@openshift-bot
Copy link

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

@openshift-ci openshift-ci bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jul 13, 2021
@slintes
Copy link
Member Author

slintes commented Jul 14, 2021

/remove-lifecycle rotten

@openshift-ci openshift-ci bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. labels Jul 14, 2021
@slintes
Copy link
Member Author

slintes commented Jul 14, 2021

/remove-lifecycle rotten

@openshift-ci openshift-ci bot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Jul 14, 2021
@openshift-bot
Copy link

Inactive enhancement proposals go stale after 28d of inactivity.

See https://github.com/openshift/enhancements#life-cycle for details.

Mark the proposal as fresh by commenting /remove-lifecycle stale.
Stale proposals rot after an additional 30d of inactivity and eventually close.
Exclude this proposal from closing by commenting /lifecycle frozen.

If this proposal is safe to close now please do so with /close.

/lifecycle stale

@openshift-ci openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 21, 2021
@openshift-bot
Copy link

Stale enhancement proposals rot after 7d of inactivity.

See https://github.com/openshift/enhancements#life-cycle for details.

Mark the proposal as fresh by commenting /remove-lifecycle rotten.
Rotten proposals close after an additional 7d of inactivity.
Exclude this proposal from closing by commenting /lifecycle frozen.

If this proposal is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

@openshift-ci openshift-ci bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Sep 28, 2021
@openshift-bot
Copy link

Rotten enhancement proposals close after 7d of inactivity.

See https://github.com/openshift/enhancements#life-cycle for details.

Reopen the proposal by commenting /reopen.
Mark the proposal as fresh by commenting /remove-lifecycle rotten.
Exclude this proposal from closing again by commenting /lifecycle frozen.

/close

@openshift-ci openshift-ci bot closed this Oct 5, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Oct 5, 2021

@openshift-bot: Closed this PR.

In response to this:

Rotten enhancement proposals close after 7d of inactivity.

See https://github.com/openshift/enhancements#life-cycle for details.

Reopen the proposal by commenting /reopen.
Mark the proposal as fresh by commenting /remove-lifecycle rotten.
Exclude this proposal from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants