Bug 1916843: collect logs from openshift-sdn-controller pod

[Serhii1011010]

sdn-controller (openshift-sdn namespace) emits important messages when it finds issues affecting Egress IPs. The important messages are the following:

“Node %s is not Ready”: A node has been set offline for egress IPs because it is reported not ready at API
“Node %s may be offline... retrying”: An egress node has failed the egress IP health check once, so it has big chances to be marked as offline soon or, at the very least, there has been a connectivity glitch.
“Node %s is offline”: An egress node has failed enough probes to have been marked offline for egress IPs. If it has egress CIDRs assigned, its egress IPs have been moved to other nodes. Indicates issues at either the node or the network between the master and the node.
“Node %s is back online”: This indicates that a node has recovered from the condition described at the previous message, by starting succeeding the egress IP health checks. Useful just in case that previous “Node %s is offline” messages are lost, so that we have a clue that there was failure previously.

As IO is gathered every 2hrs we want to gather latest occurrences of those errors in logs

Categories

• Bugfix
• Enhancement
• Backporting
• Others (CI, Infrastructure, Documentation)

Sample archive

• config/pod/openshift-sdn/logs/{pod-name}/errors.log

Documentation

• docs/gathered-data.md

Unit Tests

• pkg/gather/clusterconfig/gather_logs_test.go improved

Privacy

Yes. There are no sensitive data in the newly collected information.

Changelog

// TODO:

References

https://issues.redhat.com/browse/CCXDEV-3607
https://bugzilla.redhat.com/show_bug.cgi?id=1916843
https://access.redhat.com/solutions/???

[openshift-ci-robot]

@Sergey1011010: This pull request references Bugzilla bug 1916843, which is invalid:

• expected the bug to target the "4.7.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1916843: collect logs from openshift-sdn-controller pod

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Serhii1011010]

/bugzilla refresh

[openshift-ci-robot]

@Sergey1011010: This pull request references Bugzilla bug 1916843, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.7.0) matches configured target release for branch (4.7.0)
• bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Serhii1011010]

/retest

[openshift-ci-robot]

@Sergey1011010: This pull request references Bugzilla bug 1916843, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.7.0) matches configured target release for branch (4.7.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1916843: collect logs from openshift-sdn-controller pod

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Serhii1011010]

/retest

[tremes]

Verified locally with some other message from sdn-controller pod (we will need to update the sample archive once we have some example log). Thanks
/lgtm

[tremes]

We should be able to get some example log based on the steps mentioned in https://issues.redhat.com/browse/CCXDEV-3607. Please try to update the PR to contain the sample archive update.
/hold

[tremes]

/unhold

[tremes]

/lgtm

[tremes]

/retest

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[Serhii1011010]

/retest

[Serhii1011010]

/retest

[tremes]

I only checked the last commits.
/lgtm

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Sergey1011010, tremes

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Sergey1011010,tremes]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@Sergey1011010: All pull requests linked via external trackers have merged:

• openshift/insights-operator#314

Bugzilla bug 1916843 has been moved to the MODIFIED state.

In response to this:

Bug 1916843: collect logs from openshift-sdn-controller pod

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.