OCPBUGS-27264: Only reconcile on Node updates with Label changes

[danwinship]

e2e-aws-ovn-shared-to-local-gateway-mode-migration and its opposite flake about 50% of the time with

+ oc patch Network.operator.openshift.io cluster --type=merge --patch '{"spec":{"defaultNetwork":{"ovnKubernetesConfig":{"gatewayConfig":{"routingViaHost":false}}}}}'
network.operator.openshift.io/cluster patched
+ oc wait co network --for=condition=PROGRESSING=True --timeout=60s
error: timed out waiting for the condition on clusteroperators/network 

This is because #1721 changed CNO to re-reconcile any time any Node object changes, but Nodes change a lot (specifically, their Conditions), so now we do a ton of unnecessary reconciling, causing CNO to be super busy and lag behind in processing events. This PR fixes it to only reconcile when node labels change.

(I don't know whether CNO lagginess causes any other problems currently?)

[openshift-ci-robot]

@danwinship: This pull request references Jira Issue OCPBUGS-27264, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.16.0) matches configured target version for branch (4.16.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @anuragthehatter

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

e2e-aws-ovn-shared-to-local-gateway-mode-migration and its opposite flake about 50% of the time with

+ oc patch Network.operator.openshift.io cluster --type=merge --patch '{"spec":{"defaultNetwork":{"ovnKubernetesConfig":{"gatewayConfig":{"routingViaHost":false}}}}}'
network.operator.openshift.io/cluster patched
+ oc wait co network --for=condition=PROGRESSING=True --timeout=60s
error: timed out waiting for the condition on clusteroperators/network 

This is because #1721 changed CNO to re-reconcile any time any Node object changes, but Nodes change a lot (specifically, their Conditions), so now we do a ton of unnecessary reconciling. This PR fixes it to only reconcile when node labels change.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[danwinship]

/retest

[openshift-ci-robot]

@danwinship: This pull request references Jira Issue OCPBUGS-27264, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.16.0) matches configured target version for branch (4.16.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @anuragthehatter

Details

In response to this:

e2e-aws-ovn-shared-to-local-gateway-mode-migration and its opposite flake about 50% of the time with

+ oc patch Network.operator.openshift.io cluster --type=merge --patch '{"spec":{"defaultNetwork":{"ovnKubernetesConfig":{"gatewayConfig":{"routingViaHost":false}}}}}'
network.operator.openshift.io/cluster patched
+ oc wait co network --for=condition=PROGRESSING=True --timeout=60s
error: timed out waiting for the condition on clusteroperators/network 

This is because #1721 changed CNO to re-reconcile any time any Node object changes, but Nodes change a lot (specifically, their Conditions), so now we do a ton of unnecessary reconciling, causing CNO to be super busy and lag behind in processing events. This PR fixes it to only reconcile when node labels change.

(I don't know whether CNO lagginess causes any other problems currently?)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[danwinship]

/retest-required

[tssurya]

/lgtm
cc @wizhaoredhat
wondering if this can be further reduced to only checking for the necessary labels but this LGTM.
CI seems to be passing for the gateway mode migration jobs

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danwinship, tssurya

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [danwinship]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD 2eed7cf and 2 for PR HEAD b81354d in total

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD c793774 and 1 for PR HEAD b81354d in total

[wizhaoredhat]

LGTM

[openshift-ci]

@danwinship: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-vsphere-ovn-dualstack-primaryv6	b81354d	link	false	/test e2e-vsphere-ovn-dualstack-primaryv6

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[danwinship]

/retest-required

[openshift-ci-robot]

@danwinship: Jira Issue OCPBUGS-27264: All pull requests linked via external trackers have merged:

• openshift/cluster-network-operator#2206

Jira Issue OCPBUGS-27264 has been moved to the MODIFIED state.

Details

In response to this:

e2e-aws-ovn-shared-to-local-gateway-mode-migration and its opposite flake about 50% of the time with

+ oc patch Network.operator.openshift.io cluster --type=merge --patch '{"spec":{"defaultNetwork":{"ovnKubernetesConfig":{"gatewayConfig":{"routingViaHost":false}}}}}'
network.operator.openshift.io/cluster patched
+ oc wait co network --for=condition=PROGRESSING=True --timeout=60s
error: timed out waiting for the condition on clusteroperators/network 

This is because #1721 changed CNO to re-reconcile any time any Node object changes, but Nodes change a lot (specifically, their Conditions), so now we do a ton of unnecessary reconciling, causing CNO to be super busy and lag behind in processing events. This PR fixes it to only reconcile when node labels change.

(I don't know whether CNO lagginess causes any other problems currently?)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[danwinship]

/cherry-pick release-4.15

[openshift-cherrypick-robot]

@danwinship: new pull request created: #2212

Details

In response to this:

/cherry-pick release-4.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-bot]

[ART PR BUILD NOTIFIER]

This PR has been included in build cluster-network-operator-container-v4.16.0-202401191549.p0.ge53cc19.assembly.stream for distgit cluster-network-operator.
All builds following this will include this PR.

[openshift-merge-robot]

Fix included in accepted release 4.16.0-0.nightly-2024-01-21-092529