TRT-2426: Revert #85 " \tOCPNODE-3611: promote openshift ClusterImagePolicy to default featureset" #87
Conversation
openshift-ci-robot
added
the
jira/valid-reference
|
@smg247: This pull request references TRT-2426 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the ticket to target the "4.21.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
wking
left a comment
wking
left a comment
There was a problem hiding this comment.
openshift/origin#30318 and openshift/origin#30480 were supposed to have made the test suite comfortable with the late-update ClusterImagePolicy rollout, but I guess we still have something more to do in that space. In the meantime, we want nightlies to keep getting accepted, so:
/lgtm
…-openshift-cip"" This reverts commit 7a5dcee. This one has taken us some time: * 2025-08-27, 94f7582, openshift#82 was our first attempt at enabling the ClusterImagePolicy. * ...but it tripped up the origin test suite, so it was reverted in 2025-08-28, c40e7b9, openshift#83. * Qi then hardened the test suite with openshift/origin@d3af51e4acb (not fail upgrade checks if all nodes are ready, 2025-09-29, openshift/origin#30318) and openshift/origin@2fd0d8e242 (Upgrade test add 2min grace period allow non-drain updates to complete, 2025-11-12, openshift/origin#30480). * With the tougher CI in place, we tried a second time with 2025-11-17, 1f89a67, openshift#85. * ...but still tripped up origin, with runs like [1] taking 2.25m (more than the 2m grace period): I1119 17:26:21.890667 1511 upgrade.go:629] Waiting on pools to be upgraded I1119 17:26:21.939178 1511 upgrade.go:792] Pool master is still reporting (Updated: false, Updating: true, Degraded: false) I1119 17:26:21.939259 1511 upgrade.go:666] Invariant violation detected: master pool requires update but nodes not ready. Waiting up to 2m0s for non-draining updates to complete I1119 17:26:31.984116 1511 upgrade.go:792] Pool master is still reporting (Updated: false, Updating: true, Degraded: false) ... I1119 17:28:21.981438 1511 upgrade.go:792] Pool master is still reporting (Updated: false, Updating: true, Degraded: false) I1119 17:28:21.981514 1511 upgrade.go:673] Invariant violation detected: the "master" pool should be updated before the CVO reports available at the new version and: $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.21-upgrade-from-stable-4.20-e2e-gcp-ovn-rt-upgrade/1991158541779472384/artifacts/e2e-gcp-ovn-rt-upgrade/gather-extra/artifacts/inspect/cluster-scoped-resources/machineconfiguration.openshift.io/machineconfigpools/master.yaml | yaml2json | jq -r '.status.conditions[] | select(.type == "Updating") | .lastTransitionTime + " " + .status' 2025-11-19T17:28:36Z False 28:36 - 26:21 = 135s = 2.25m, which overshot the 2m grace period. The second attempt was reverted in 7a5dcee, openshift#87. * Qi then hardened the test suite further with openshift/origin@c17e560263 (Update grace period for cluster upgrade to 10 minutes, 2025-11-19, #openshift/origin#30506). * This commit is taking a third attempt at enabling the ClusterImagePolicy. [1]: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.21-upgrade-from-stable-4.20-e2e-gcp-ovn-rt-upgrade/1991158541779472384
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jupierce, smg247, wking The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
Still needs |
|
/retest-required |
|
/verified by CI Plenty of Services should be rejected for evicted pods failures outside of this pr |
|
@neisw: Overrode contexts on behalf of neisw: ci/prow/e2e-aws In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
openshift-ci-robot
added
the
verified
|
@neisw: This PR has been marked as verified by In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@smg247: all tests passed! Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
5 participants
Reverts #85 ; tracked by TRT-2426
Per OpenShift policy, we are reverting this breaking change to get CI and/or nightly payloads flowing again.
Began failing in https://amd64.ocp.releases.ci.openshift.org/releasestream/4.21.0-0.nightly/release/4.21.0-0.nightly-2025-11-19-020220 on aggregated-gcp-ovn-rt-upgrade-4.21-minor and aggregated-aws-ovn-upgrade-4.21-micro-fips
To unrevert this, revert this PR, and layer an additional separate commit on top that addresses the problem. Before merging the unrevert, please run these jobs on the PR and check the result of these jobs to confirm the fix has corrected the problem:
CC: @QiWang19