OCPNODE-3203: Add a test suite for kubeletconfig testing

[ngopalak-redhat]

This PR introduces a new test suite specifically for node component testing, dubbed "Long Running Tests."

Following discussions with the MCO team on Slack, we agreed to separate specific disruptive node tests from the main MCO disruptive test suite.

Why not the Serial Suite? These tests cannot be part of the existing serial suite because they require multiple node reboots (exceeding the 3-restart limit) and have a significant runtime duration.

Goal is to establish a dedicated suite for tests that are disruptive and time-consuming but critical for release verification of the node component configuration (Kubelet).

Implementation Details

Adds the framework for long-running tests. Currently includes one test case: Changing the Kubelet Log Level.

• The test applies a Kubelet config change.
• It explicitly waits for the node reboot cycle to complete.

Configs are applied to a single node rather than all three. This reduces the blast radius and significantly speeds up execution. The suite includes logic to revert changes and clean up the node state post-execution.

Future Work / Roadmap

• Post-merge, I will add tests for system-compressible and auto-node-sizing.

• I will work with the MCO team to configure this as a Periodic Job (similar to the MCO disruptive tests). These tests will not run on every PR but will be a Release Blocking requirement.

Sample run: https://gist.github.com/ngopalak-redhat/0c63bddf63a0a49c46c9dd2a13fad465

[openshift-ci-robot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci-robot]

@ngopalak-redhat: This pull request references OCPNODE-3203 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@ngopalak-redhat: This pull request references OCPNODE-3203 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

This PR introduces a new test suite specifically for node component testing, dubbed "Long Running Tests."

Following discussions with the MCO team on Slack, we agreed to separate specific disruptive node tests from the main MCO disruptive test suite.

Why not the Serial Suite? These tests cannot be part of the existing serial suite because they require multiple node reboots (exceeding the 3-restart limit) and have a significant runtime duration.

Goal is to establish a dedicated suite for tests that are disruptive and time-consuming but critical for release verification of the node component configuration (Kubelet).

Implementation Details

Adds the framework for long-running tests. Currently includes one test case: Changing the Kubelet Log Level.

• The test applies a Kubelet config change.
• It explicitly waits for the node reboot cycle to complete.

Configs are applied to a single node rather than all three. This reduces the blast radius and significantly speeds up execution. The suite includes logic to revert changes and clean up the node state post-execution.

Future Work / Roadmap

• Post-merge, I will add tests for system-compressible and auto-node-sizing.

• I will work with the MCO team to configure this as a Periodic Job (similar to the MCO disruptive tests). These tests will not run on every PR but will be a Release Blocking requirement.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[ngopalak-redhat]

/test all

[openshift-ci-robot]

@ngopalak-redhat: This pull request references OCPNODE-3203 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

This PR introduces a new test suite specifically for node component testing, dubbed "Long Running Tests."

Following discussions with the MCO team on Slack, we agreed to separate specific disruptive node tests from the main MCO disruptive test suite.

Why not the Serial Suite? These tests cannot be part of the existing serial suite because they require multiple node reboots (exceeding the 3-restart limit) and have a significant runtime duration.

Goal is to establish a dedicated suite for tests that are disruptive and time-consuming but critical for release verification of the node component configuration (Kubelet).

Implementation Details

Adds the framework for long-running tests. Currently includes one test case: Changing the Kubelet Log Level.

• The test applies a Kubelet config change.
• It explicitly waits for the node reboot cycle to complete.

Configs are applied to a single node rather than all three. This reduces the blast radius and significantly speeds up execution. The suite includes logic to revert changes and clean up the node state post-execution.

Future Work / Roadmap

• Post-merge, I will add tests for system-compressible and auto-node-sizing.

• I will work with the MCO team to configure this as a Periodic Job (similar to the MCO disruptive tests). These tests will not run on every PR but will be a Release Blocking requirement.

Sample run: https://gist.github.com/ngopalak-redhat/0c63bddf63a0a49c46c9dd2a13fad465

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[ngopalak-redhat]

/test all

[openshift-ci-robot]

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

[ngopalak-redhat]

@cpmeadors Please review

[qiliRedHat]

@ngopalak-redhat In tests that also need to config kubeletconfig or wait for machineconfig update, some of your functions in the kubeletconfig_features.go file can be reused. Should we consider to put the common functions in a utils.go or node_utils.go? CC: @cpmeadors

[cpmeadors]

/lgtm

[cpmeadors]

/assigne @neisw

[cpmeadors]

/assign @dgoodwin

[dgoodwin]

I was hoping we would end up with a general purpose disruptive suite as this comes up for multiple teams quite often. Is there a reason this could not be morphed to be so? Long running is still fine I think as our ability to run very long jobs is now improved last I heard.

Out of curiosity how long are we talking for your current tests?

[cpmeadors]

I was hoping we would end up with a general purpose disruptive suite as this comes up for multiple teams quite often. Is there a reason this could not be morphed to be so? Long running is still fine I think as our ability to run very long jobs is now improved last I heard.

This is a step in that direction. We wanted to get it working first, then adapt for more general usage.

Out of curiosity how long are we talking for your current tests?

@ngopalak-redhat can you answer this?

[dgoodwin]

I'd probably prefer we just name this to be what we want right from the get-go, rather than leaving yourselves a task to come back to later. Those kinds of things tend to not happen sometimes, but we could keep it kinda quiet while you get it where you want, then announce more broadly? I can avoid pointing anyone to it until you give the go-ahead.

[dgoodwin]

To further the point, the test names would need renaming because they contain the suite, which technically bloats all the dbs and should be reflected by a rename in the component-mapping repo, which is a pain. Job renames lose history as well.
I would encourage getting this right the first time both in suite, test, and job naming.

[ngopalak-redhat]

@dgoodwin / @cpmeadors I have chosen a generalized name for the test suite: openshift/disruptive-longrunning.

Currently, the test I added runs serially and takes about 5 minutes. However, the Node team plans to add more comprehensive tests that may take up to 15 minutes and require multiple node restarts. Thinking of merging usernamespace test (https://github.com/openshift/origin/blob/main/test/extended/node/nested_container.go) also into this one.

For example, the AutoSizingReserved feature (which you are aware of) requires multiple restarts to verify that the enable/disable logic functions correctly and that values are applied properly.

[openshift-ci-robot]

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

[dgoodwin]

/approve

[ngopalak-redhat]

/verified by @ngopalak-redhat
/retest-required

[openshift-ci-robot]

@ngopalak-redhat: This PR has been marked as verified by @ngopalak-redhat.

Details

In response to this:

/verified by @ngopalak-redhat
/retest-required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[dgoodwin]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cpmeadors, dgoodwin, ngopalak-redhat

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [dgoodwin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@ngopalak-redhat: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.