OCPEDGE-2280: Add Adaptable Topology, reorganize topology enhancements #1905
Conversation
openshift-ci-robot
added
the
jira/valid-reference
|
@jaypoulz: This pull request explicitly references no jira issue. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
jaypoulz
force-pushed
the
adaptable-topology
branch
from
3aa08a3 to
40d4ff0
Compare
|
/retitle OCPEDGE-2280: Add Adaptable Topology, reorganize topology enhancements |
openshift-ci
bot
changed the title
|
@jaypoulz: This pull request references OCPEDGE-2280 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target either version "4.21." or "openshift-4.21.", but it targets "openshift-4.22" instead. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/retest |
Introduce Adaptable topology, a cluster-topology mode that adjusts control-plane and infrastructure behavior based on node count. Clusters can install with Adaptable topology or transition from SingleReplica as a Day 2 operation. Key components: - Infrastructure API adds Adaptable enum value - Operator compatibility annotations for in-payload and OLM operators - Library-go utilities for node count awareness - cluster-etcd-operator enhanced scaling logic for 2↔3 nodes - oc adm topology transition CLI command with compatibility checks - Console marketplace filtering and compatibility display - Initial Topology Audit covering 39 operators The initial implementation supports SingleReplica-to-Adaptable transitions. Future stages will add AutomaticQuorumRecovery (AQR) for DualReplica behavior on two-node configurations. Reviewers assigned across 18 teams including control plane, OLM, API, installer, console, monitoring, and core operator teams. This also reorganizes edge topology enhancements under a new edge-topologies/ directory: - edge-topologies/adaptable-topology.md (new) - edge-topologies/single-node/ (moved) - edge-topologies/two-node/ (moved and reorganized) - two-node-fencing.md (renamed from tnf.md) - two-node-arbiter.md (renamed from arbiter-clusters.md)
jaypoulz
force-pushed
the
adaptable-topology
branch
from
40d4ff0 to
6e81004
Compare
|
@jaypoulz: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
| Adaptable topology is enabled | ||
| - Provide a subscription function for topology changes that operators invoke | ||
| to enable Adaptable topology behavior | ||
| - Subscribe to node count changes and provide notifications when |
There was a problem hiding this comment.
It seems like a more natural way of triggering operators to make a change relevant to cluster changes might be to utilize the controller runtime events library for notifications (and as a result allowing operators to reconcile the changes)
We might be able to utilize the GenericEvent but I think ideally it would make sense to create a TopologyTransitionEvent type specific for these transitions
| 1. The cluster administrator adds a new control-plane node to the cluster | ||
| 2. The node joins the cluster and receives the control-plane labels | ||
| 3. Operators watching the Infrastructure config detect the node count change | ||
| 4. When crossing the 2→3 control-plane node threshold: |
There was a problem hiding this comment.
We also support >3 control plane nodes, e.g. 4 or 5 etcd members. I think this etcd scaling process needs to be done then, too.
| 4. When crossing the 2→3 control-plane node threshold: | ||
| - CEO initiates the etcd scaling process, starting learner instances on the two control-plane nodes not running etcd | ||
| - CEO coordinates an atomic transition to promote both learner instances together | ||
| - Other operators adjust their behavior to match HighlyAvailable control-plane topology |
There was a problem hiding this comment.
would that happen concurrently with CEO, or after? As CEO / etcd scaling is high risk operation, I would feel better to first do that, and then somehow trigger the other operators.
| - CEO initiates the etcd scaling process, starting learner instances on the two control-plane nodes not running etcd | ||
| - CEO coordinates an atomic transition to promote both learner instances together | ||
| - Other operators adjust their behavior to match HighlyAvailable control-plane topology | ||
| 5. When scaling down and crossing the 3→2 control-plane node threshold: |
There was a problem hiding this comment.
what is triggering this scaling down? Is it a simple "oc delete node/cp-1"? How then ensure atomicity?
I think we need an "oc adm topology" command to do that.
We also need to cross check with the procedure we have in place to replace a failed control plan node. If that involves first removing the node, we probably do not want to trigger the etcd
| This prevents invalid configurations where control-plane and infrastructure | ||
| topologies are out of sync. | ||
|
|
||
| #### Shared Utilities in library-go |
There was a problem hiding this comment.
This still feels like an awkward way of getting and watch the status. Why not have a new attribute "effectiveTopology" which can change and shows the current topology?
Also, for cluster admins, its not easy to see what the current effective topolog is, e.g. during the control plane scaling process. We need to have a status field that shows e.g. "etcd cluster extension in progress, adding learners,....."
|
|
||
| **Topology Mismatch**: The cluster configuration would claim HighlyAvailable | ||
| while still operating as SingleReplica until nodes are provisioned. | ||
| The declared topology differs from actual behavior. |
There was a problem hiding this comment.
Adaptable will have this intermediate state, too. E.g. during etcd learner phase when transitioning from SNO to MNO
| The declared topology differs from actual behavior. | ||
|
|
||
| **Complex Support Matrix**: Supporting arbitrary topology transitions creates | ||
| a complex matrix. |
There was a problem hiding this comment.
But does not Adaptable also allow arbitrary tranistions, by adding/removing nodes? This will create a complex matrix, too.
|
|
||
| The one-way transition model simplifies testing. | ||
| We only test transitions into Adaptable topology and can stagger support | ||
| for different source topologies. |
There was a problem hiding this comment.
but we also need to test scaleout/scaledown with adaptable, which creates tons of possible paths we need to validate. Or not?
4 participants
This enhancement introduces Adaptable topology, a new cluster-topology mode that enables clusters to dynamically adjust their behavior based on node count. This allows SingleReplica clusters to scale to multi-node configurations without redeployment.
Key features:
The proposal includes complete workflow descriptions, API extensions, test plans, and version skew strategy. Future stages will add AutomaticQuorumRecovery (AQR) to enable DualReplica-based resiliency for two-node configurations.