Bug 1995913: Degraded status in the OCM controller #486

tremes · 2021-08-16T14:23:21Z

This adds status checking to the OCM controller. If the response from the OCM API is an HTTP error other than 404 (which means that SCA certs are not allowed for the given organization) then try again with exponential backoff and after few attempts mark the operator as degraded.
At the same time, the status doesn't have to be changed to degraded in a disconnected environment. This means that any non-HTTP error is not considered as a reason for marking the operator as degraded.

Sample Archive

No new data

Documentation

No documentation update

Unit Tests

No new test

Privacy

Yes. There are no sensitive data in the newly collected information.

Changelog

Breaking Changes

No

References

https://issues.redhat.com/browse/???
https://bugzilla.redhat.com/show_bug.cgi?id=1995913
https://access.redhat.com/solutions/???

tremes · 2021-08-18T06:38:53Z

pkg/ocm/ocm.go

+	if err != nil {
+		return nil, fmt.Errorf("timed out waiting for the successful response from %s", endpoint)
+	}
+	return data, nil


This method is little bit complicated (especially due the fact that 404 and offline env. are not considered as errors from the "degraded" perspective) compared to counting the threshold in the controllerstatus and the checking it in the status.go (similarly as here)

The approach with exp. backoff is more "agressive":

if HTTP error other than 404 then try again in 5 min (if this is still true then try again in 10 min, 20min, 40min => mark as degraded)

Compared to the other approach:

if HTTP error other than 404 then only increase the count and do nothing

try again in default period (which is 8h in this case) if there's still an error then increase the count again

if count higher than some threshold (let's say 2 or 3) then mark IO as degraded

WIth the first approach we can go degraded in ~1h15min, with the second one it would be after 16h-24h. I am not sure what is better TBH. The certs are valid for a long time (1 year by default), but on the other hand, if the content changes then you may want to update the certs quickly.

I updated the initial backoff step to c.configurator.Config().OCMConfig.Interval / 32 (that is 15 minutes by default) so the times are bit longer than mentioned above.

tremes · 2021-08-18T07:29:34Z

/test insights-operator-e2e-tests

pkg/controller/status/status.go

tremes · 2021-08-20T07:01:18Z

/retest

openshift-ci · 2021-08-20T07:01:37Z

@tremes: This pull request references Bugzilla bug 1995913, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.9.0) matches configured target release for branch (4.9.0)
bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

No GitHub users were found matching the public email listed for the QA contact in Bugzilla (dmisharo@redhat.com), skipping review request.

In response to this:

Bug 1995913: Degraded status in the OCM controller

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2021-08-20T07:01:41Z

@tremes: This pull request references Bugzilla bug 1995913, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.9.0) matches configured target release for branch (4.9.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

No GitHub users were found matching the public email listed for the QA contact in Bugzilla (dmisharo@redhat.com), skipping review request.

In response to this:

Bug 1995913: Degraded status in the OCM controller

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tremes · 2021-08-23T06:46:04Z

/retest

tremes · 2021-08-25T05:54:22Z

/retest

rluders · 2021-08-26T11:32:11Z

Tested and it seems good.
/lgtm

openshift-ci · 2021-08-26T11:32:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rluders, tremes

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [rluders,tremes]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2021-08-26T12:39:53Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-26T12:52:52Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T01:27:25Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T01:40:24Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T04:02:52Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T04:15:50Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T05:59:50Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T06:59:49Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T14:35:52Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T14:47:52Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T15:11:51Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T15:23:50Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T16:59:51Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T19:11:51Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T19:24:52Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

tremes · 2021-08-29T11:50:43Z

/test insights-operator-e2e-tests

openshift-bot · 2021-08-30T06:03:51Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-30T07:27:53Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-30T08:27:52Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-30T10:15:55Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-30T10:27:52Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-30T17:48:06Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-30T20:10:53Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-30T21:02:54Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-30T22:59:54Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-31T11:15:53Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-31T12:20:55Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

iNecas · 2021-08-31T15:04:58Z

/test insights-operator-e2e-tests

iNecas · 2021-08-31T15:06:35Z

/retest

openshift-bot · 2021-08-31T15:55:04Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2021-08-31T16:35:48Z

@tremes: All pull requests linked via external trackers have merged:

openshift/insights-operator#486

Bugzilla bug 1995913 has been moved to the MODIFIED state.

In response to this:

Bug 1995913: Degraded status in the OCM controller

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tremes added 2 commits August 16, 2021 16:17

Mark as degraded when the pulling of SCA certs failed

a797604

Degraded with exponential backoff

bc4845f

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 16, 2021

openshift-ci bot requested review from smarterclayton and tisnik August 16, 2021 14:23

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 16, 2021

tremes mentioned this pull request Aug 16, 2021

Bug 1988308: Add feature for Insights operator pulling SCA certs openshift/api#983

Merged

Update the error cases and add some logging

7368993

tremes changed the title ~~WIP: Degraded status in the OCM controller~~ Degraded status in the OCM controller Aug 17, 2021

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 17, 2021

tremes commented Aug 18, 2021

View reviewed changes

tremes added 2 commits August 19, 2021 08:10

Update the backoff a little

2a827dd

Fix

5044445

tremes commented Aug 19, 2021

View reviewed changes

pkg/controller/status/status.go Outdated Show resolved Hide resolved

Remove unneccesary block

dd1d210

tremes changed the title ~~Degraded status in the OCM controller~~ Bug 1995913: Degraded status in the OCM controller Aug 20, 2021

openshift-ci bot added bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Aug 20, 2021

openshift-ci bot assigned rluders Aug 26, 2021

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 26, 2021

openshift-merge-robot merged commit d97e3c8 into openshift:master Aug 31, 2021

Bug 1995913: Degraded status in the OCM controller #486

Bug 1995913: Degraded status in the OCM controller #486

Conversation

tremes commented Aug 16, 2021 • edited Loading

Categories

Sample Archive

Documentation

Unit Tests

Privacy

Changelog

Breaking Changes

References

tremes Aug 18, 2021

Choose a reason for hiding this comment

tremes Aug 20, 2021

Choose a reason for hiding this comment

tremes commented Aug 18, 2021

tremes commented Aug 20, 2021

openshift-ci bot commented Aug 20, 2021

openshift-ci bot commented Aug 20, 2021

tremes commented Aug 23, 2021

tremes commented Aug 25, 2021

rluders commented Aug 26, 2021

openshift-ci bot commented Aug 26, 2021

openshift-bot commented Aug 26, 2021

openshift-bot commented Aug 26, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

tremes commented Aug 29, 2021

openshift-bot commented Aug 30, 2021

openshift-bot commented Aug 30, 2021

openshift-bot commented Aug 30, 2021

openshift-bot commented Aug 30, 2021

openshift-bot commented Aug 30, 2021

openshift-bot commented Aug 30, 2021

openshift-bot commented Aug 30, 2021

openshift-bot commented Aug 30, 2021

openshift-bot commented Aug 30, 2021

openshift-bot commented Aug 31, 2021

openshift-bot commented Aug 31, 2021

iNecas commented Aug 31, 2021

iNecas commented Aug 31, 2021

openshift-bot commented Aug 31, 2021

openshift-ci bot commented Aug 31, 2021

tremes commented Aug 16, 2021 •

edited

Loading