wip

[QiWang19]

No description provided.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: QiWang19
Once this PR has been reviewed and has the lgtm label, please assign stbenjam for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[QiWang19]

/testwith openshift/cluster-update-keys/main/e2e-aws-upgrade openshift/cluster-update-keys#85 #30475

[QiWang19]

/testwith openshift/cluster-update-keys/main/e2e-aws-upgrade https://github.com/openshift/cluster-update-keys/pull/85 https://github.com/openshift/origin/pull/30475

https://prow.ci.openshift.org/view/gs/test-platform-results/logs/multi-pr-openshift-origin-30475-openshift-cluster-update-keys-85-openshift-origin-30475-e2e-aws-upgrade/1988291772496744448

I1111 18:54:23.282898 1701 upgrade.go:601] Completed upgrade to registry.build05.ci.openshift.org/ci-op-zf6ppfz5/release@sha256:bca5c95fb7b3d076fdaa7b83d83df753214bfb8a31479e6b7a0ec54d02dfa94a
I1111 18:54:23.292309 1701 upgrade.go:626] Waiting on pools to be upgraded
I1111 18:54:23.298949 1701 upgrade.go:684] Not all nodes are ready. Total nodes: 6, Not ready nodes: 2
I1111 18:54:23.298977 1701 upgrade.go:686]   - node/ip-10-0-6-117.ec2.internal: MCD state="Working" (expected 'Done')
I1111 18:54:23.298988 1701 upgrade.go:686]   - node/ip-10-0-60-139.ec2.internal: MCD state="Working" (expected 'Done')
I1111 18:54:23.308304 1701 upgrade.go:843] Pool master is still reporting (Updated: false, Updating: true, Degraded: false)
I1111 18:54:23.308357 1701 upgrade.go:710] Invariant violation detected: the "master" pool should be updated before the CVO reports available at the new version
I1111 18:54:23.308380 1701 upgrade.go:711]   - allNodesReady: false
I1111 18:54:23.308404 1701 upgrade.go:712]   - master pool requiresUpdate: true
I1111 18:54:23.308422 1701 upgrade.go:713]   - Not ready nodes details:
I1111 18:54:23.308442 1701 upgrade.go:715]     * node/ip-10-0-6-117.ec2.internal: MCD state="Working" (expected 'Done')
I1111 18:54:23.308460 1701 upgrade.go:715]     * node/ip-10-0-60-139.ec2.internal: MCD state="Working" (expected 'Done')
I1111 18:54:23.312520 1701 upgrade.go:722]   - Master pool conditions: [map[lastTransitionTime:2025-11-11T17:53:18Z message: reason: status:False type:PinnedImageSetsDegraded] map[lastTransitionTime:2025-11-11T17:53:20Z message: reason: status:False type:RenderDegraded] map[lastTransitionTime:2025-11-11T18:43:22Z message: reason: status:False type:NodeDegraded] map[lastTransitionTime:2025-11-11T18:43:22Z message: reason: status:False type:Degraded] map[lastTransitionTime:2025-11-11T18:54:16Z message: reason: status:False type:Updated] map[lastTransitionTime:2025-11-11T18:54:16Z message:All nodes are updating to MachineConfig rendered-master-4e98031016fb07fc66e0adfe1234b365 reason: status:True type:Updating]]
I1111 18:54:23.317542 1701 upgrade.go:843] Pool worker is still reporting (Updated: false, Updating: true, Degraded: false)
I1111 18:54:33.327311 1701 upgrade.go:684] Not all nodes are ready. Total nodes: 6, Not ready nodes: 2
I1111 18:54:33.327345 1701 upgrade.go:686]   - node/ip-10-0-6-117.ec2.internal: MCD state="Working" (expected 'Done')
I1111 18:54:33.327359 1701 upgrade.go:686]   - node/ip-10-0-60-139.ec2.internal: MCD state="Working" (expected 'Done')
I1111 18:54:33.342477 1701 upgrade.go:843] Pool master is still reporting (Updated: false, Updating: true, Degraded: false)
I1111 18:54:33.346763 1701 upgrade.go:843] Pool worker is still reporting (Updated: false, Updating: true, Degraded: false)
I1111 18:54:43.326847 1701 upgrade.go:684] Not all nodes are ready. Total nodes: 6, Not ready nodes: 1
I1111 18:54:43.326884 1701 upgrade.go:686]   - node/ip-10-0-60-139.ec2.internal: MCD state="Working" (expected 'Done')
I1111 18:54:43.335344 1701 upgrade.go:843] Pool master is still reporting (Updated: false, Updating: true, Degraded: false)
I1111 18:54:43.339915 1701 upgrade.go:843] Pool worker is still reporting (Updated: false, Updating: true, Degraded: false)
I1111 18:54:53.328319 1701 upgrade.go:684] Not all nodes are ready. Total nodes: 6, Not ready nodes: 1
I1111 18:54:53.328347 1701 upgrade.go:686]   - node/ip-10-0-9-230.ec2.internal: MCD state="Working" (expected 'Done')
I1111 18:54:53.341509 1701 upgrade.go:843] Pool master is still reporting (Updated: false, Updating: true, Degraded: false)
I1111 18:54:53.350743 1701 upgrade.go:843] Pool worker is still reporting (Updated: false, Updating: true, Degraded: false)
I1111 18:55:03.326617 1701 upgrade.go:684] Not all nodes are ready. Total nodes: 6, Not ready nodes: 2
I1111 18:55:03.326653 1701 upgrade.go:686]   - node/ip-10-0-17-83.ec2.internal: MCD state="Working" (expected 'Done')
I1111 18:55:03.326665 1701 upgrade.go:686]   - node/ip-10-0-9-230.ec2.internal: MCD state="Working" (expected 'Done')
I1111 18:55:03.337489 1701 upgrade.go:843] Pool master is still reporting (Updated: false, Updating: true, Degraded: false)
I1111 18:55:03.342683 1701 upgrade.go:843] Pool worker is still reporting (Updated: false, Updating: true, Degraded: false)
I1111 18:55:13.326736 1701 upgrade.go:684] Not all nodes are ready. Total nodes: 6, Not ready nodes: 2
I1111 18:55:13.326772 1701 upgrade.go:686]   - node/ip-10-0-17-83.ec2.internal: MCD state="Working" (expected 'Done')
I1111 18:55:13.326786 1701 upgrade.go:686]   - node/ip-10-0-9-230.ec2.internal: MCD state="Working" (expected 'Done')
I1111 18:55:13.342348 1701 upgrade.go:843] Pool master is still reporting (Updated: false, Updating: true, Degraded: false)
I1111 18:55:13.346894 1701 upgrade.go:843] Pool worker is still reporting (Updated: false, Updating: true, Degraded: false)
I1111 18:55:23.327082 1701 upgrade.go:684] Not all nodes are ready. Total nodes: 6, Not ready nodes: 2
I1111 18:55:23.327131 1701 upgrade.go:686]   - node/ip-10-0-17-83.ec2.internal: MCD state="Working" (expected 'Done')
I1111 18:55:23.327145 1701 upgrade.go:686]   - node/ip-10-0-79-206.ec2.internal: MCD state="Working" (expected 'Done')
I1111 18:55:23.336467 1701 upgrade.go:843] Pool master is still reporting (Updated: false, Updating: true, Degraded: false)
I1111 18:55:23.347662 1701 upgrade.go:843] Pool worker is still reporting (Updated: false, Updating: true, Degraded: false)
I1111 18:55:33.327701 1701 upgrade.go:684] Not all nodes are ready. Total nodes: 6, Not ready nodes: 2
I1111 18:55:33.327738 1701 upgrade.go:686]   - node/ip-10-0-125-59.ec2.internal: MCD state="Working" (expected 'Done')
I1111 18:55:33.327751 1701 upgrade.go:686]   - node/ip-10-0-79-206.ec2.internal: MCD state="Working" (expected 'Done')
I1111 18:55:33.340120 1701 upgrade.go:843] Pool master is still reporting (Updated: false, Updating: true, Degraded: false)
I1111 18:55:33.345780 1701 upgrade.go:843] Pool worker is still reporting (Updated: false, Updating: true, Degraded: false)
I1111 18:55:42.644575 1701 adminack.go:88] Skipping admin ack test. Admin ack is not in this baseline or contains no gates.
I1111 18:55:43.326434 1701 upgrade.go:684] Not all nodes are ready. Total nodes: 6, Not ready nodes: 2
I1111 18:55:43.326467 1701 upgrade.go:686]   - node/ip-10-0-125-59.ec2.internal: MCD state="Working" (expected 'Done')
I1111 18:55:43.326476 1701 upgrade.go:686]   - node/ip-10-0-79-206.ec2.internal: MCD state="Working" (expected 'Done')
I1111 18:55:43.338349 1701 upgrade.go:843] Pool master is still reporting (Updated: false, Updating: true, Degraded: false)
I1111 18:55:43.347081 1701 upgrade.go:843] Pool worker is still reporting (Updated: false, Updating: true, Degraded: false)
I1111 18:55:53.325999 1701 upgrade.go:684] Not all nodes are ready. Total nodes: 6, Not ready nodes: 1
I1111 18:55:53.326054 1701 upgrade.go:686]   - node/ip-10-0-125-59.ec2.internal: MCD state="Working" (expected 'Done')
I1111 18:55:53.336404 1701 upgrade.go:843] Pool master is still reporting (Updated: false, Updating: true, Degraded: false)
I1111 18:56:03.325785 1701 upgrade.go:684] Not all nodes are ready. Total nodes: 6, Not ready nodes: 1
I1111 18:56:03.325817 1701 upgrade.go:686]   - node/ip-10-0-125-59.ec2.internal: MCD state="Working" (expected 'Done')
I1111 18:56:03.335800 1701 upgrade.go:843] Pool master is still reporting (Updated: false, Updating: true, Degraded: false)
I1111 18:56:13.324095 1701 upgrade.go:689] All 6 nodes are ready
I1111 18:56:13.341070 1701 upgrade.go:731] All pools completed upgrade