CORS-3687: Enhancement proposal for setting EIPs for Ingress Controller via installer #1688

miheer · 2024-09-26T03:51:38Z

Enhancement proposal for setting EIPs for Ingress Controller via installer

openshift-ci-robot · 2024-09-26T03:51:42Z

@miheer: This pull request references CORS-3687 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2024-09-26T03:51:52Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign candita for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

enhancements/ingress/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

miheer · 2024-09-26T05:31:02Z

@mtulio @patrickdillon @r4f4 PTAL

openshift-ci-robot · 2024-09-26T05:33:06Z

@miheer: This pull request references CORS-3687 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set.

In response to this:

Enhancement proposal for setting EIPs for Ingress Controller via installer

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

r4f4 · 2024-09-30T07:24:00Z

enhancements/ingress/aws-lb-eip-selection.md

+platform:
+  aws:
+    region: <AWS region>
+    lbType: NLB


Does the installer have to validate the lbType is NLB when eipAllocations are specified? What happens when lbType: Classic?

Yes we can check this. I will add a CEL.

+1 to this question, I don't see this in the EP yet.

Also, the installer doesn't use CEL, but I think it's fine to provide it in this proposal as validation guidelines.

r4f4 · 2024-09-30T07:25:36Z

enhancements/ingress/aws-lb-eip-selection.md

+  aws:
+    region: <AWS region>
+    lbType: NLB
+    networkLoadBalancer:


Suggested change

networkLoadBalancer:

ingressNetworkLoadBalancer:

So it's not confused with the one created by the Installer.

OK I will make that change.

changes made

sadasu · 2024-10-03T22:55:52Z

enhancements/ingress/aws-lb-eip-selection.md

+```
+
+In the `IngressController` status, check the status for the following:
+- Error messages for invalid eips or eips not present in the subnet of the VPC is `The service-controller component is reporting SyncLoadBalancerFailed events like: Error syncing load balancer: failed to ensure load balancer: error creating load balancer: "AllocationIdNotFound:` for status type `LoadBalancerReady` and `Available`.


When this error occurs, how does the customer/user remedy the situation? Restart installation with correct EIPs or can the user edit the ingressNetworkLoadBalancer list within the Ingress Config spec so that CCM can successfully reconcile the creation of the Ingress LB or both? Any preference?

@sadasu can you please check point 3 under section ### Implementation Details/Notes/Constraints

Please let me know WYT.

Also if an error is invoked we can either stop or ask for the correct input from user. I need to check the installer code if we can add a loop.

…er via installer

miheer · 2024-10-21T07:02:39Z

@mtulio @patrickdillon @r4f4 @sadasu @gcs278 @Miciah PTAL especially the validation part. I need help installer team to review the point 3 under section ### Implementation Details/Notes/Constraints

openshift-ci · 2024-10-21T07:14:59Z

@miheer: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

r4f4 · 2024-10-21T07:53:28Z

enhancements/ingress/aws-lb-eip-selection.md

+(i.e. any subnet without another cluster's kubernetes.io/cluster/<cluster-id> tag). 
+We can call this Predicted LB Subnet Count.
+
+We can examine the following scenarios:


Can Predicted LB Subnet count < BYO Subnet count? If not, these are the possible scenarios:

EIP Allocations count < BYO Subnet count: error

EIP Allocations count < Predicted LB Subnet count: error

Is is an issue if extra EIP Allocations are supplied?

the count of eip allocation should exactly match the number of subnets. It can't be less or greater than and must be equal to.

gcs278

Still working on reviewing. Only a couple of comments for now, will come back later.

gcs278 · 2024-10-24T03:26:57Z

enhancements/ingress/aws-lb-eip-selection.md

+creation-date: 2024-05-29
+last-updated: 2024-09-04
+tracking-link:
+  - https://issues.redhat.com/browse/CORS-3440
+see-also:
+  - "enhancements/ingress/lb-subnet-selection-aws.md"
+replaces:
+  - "enhancements/installer/aws-customer-provided-subnets.md"
+superseded-by:


I think you forgot to update some of the fields after a copy/paste

Suggested change

creation-date: 2024-05-29

last-updated: 2024-09-04

tracking-link:

- https://issues.redhat.com/browse/CORS-3440

see-also:

- "enhancements/ingress/lb-subnet-selection-aws.md"

replaces:

- "enhancements/installer/aws-customer-provided-subnets.md"

superseded-by:

creation-date: 2024-?

last-updated: 2024-?

tracking-link:

- https://issues.redhat.com/browse/CORS-3687

see-also:

replaces:

superseded-by:

gcs278 · 2024-10-24T03:29:23Z

enhancements/ingress/aws-lb-eip-selection.md

+
+This enhancement extends the OpenShift Installer's install-config, enabling cluster admins to
+configure EIPs for AWS NLB load balancer created for their default NLB IngressController at install time.
+This proposal allows the install-time configuration of subnets for the `default` IngressController.


Suggested change

This proposal allows the install-time configuration of subnets for the `default` IngressController.

gcs278 · 2024-10-24T03:31:17Z

enhancements/ingress/aws-lb-eip-selection.md

+## Motivation
+
+### User Stories
+- As a cluster administrator using installer, I want to configure default NLB IngressController to use EIPs.


So you just defined the default IngressController above, but then don't use the backticks. Just want to make sure indeed talking about the default IngressController.

Suggested change

- As a cluster administrator using installer, I want to configure default NLB IngressController to use EIPs.

- As a cluster administrator using installer, I want to configure `default` NLB IngressController to use EIPs.

gcs278 · 2024-10-24T03:34:18Z

enhancements/ingress/aws-lb-eip-selection.md

+Here's one recommended option:
+
+The Installer should count all LB subnets by predicting what subnets be chosen by the AWS CCM 
+(i.e. any subnet without another cluster's kubernetes.io/cluster/<cluster-id> tag). 


Not only subnets without another cluster's kubernetes.io/cluster/<cluster-id> tag, but the subnet won't be selected if the load balancer is external and the subnet is private.

See https://github.com/openshift/enhancements/pull/1634/files#diff-ffcfdf0d21ba360a17e0ac9846e83eec8ecf0ba8b15d429d42e7de2dbd0bfaf7R109-R116

gcs278 · 2024-10-24T03:35:49Z

enhancements/ingress/aws-lb-eip-selection.md

+           which uses the value from the field `eipAllocations` of `IngressController` CR.
+
+2. #### Validation on installer when installing in managed VPC (full-automated) based in the discovered zones used to create the cluster.
+     We will be comparing the number of Availability Zones in the region to the number of eipAllocations passed in the `install-config.yaml`.


Probably worth an explanation why: because the cluster will select 1 subnet per AZ, and the number of EIPs must be equal to subnets.

gcs278 · 2024-10-24T04:27:34Z

enhancements/ingress/aws-lb-eip-selection.md

+
+We can examine the following scenarios:
+
+##### BYO Subnet Count != EIPs Allocations && BYO Subnet Count == Predicted LB Subnet count && Predicted LB Subnet count != EIPs Allocations:


We might be able to eliminate all of this complex validation if:

We agree to simplify the new subnets API (CORS-3440: IngressController subnet selection at installation #1634 (comment))

EIP Allocations waits for new subnet API to be available and is dependent on it

But it's a tough call. Maybe we have to do this validation in the beginning so you can release the EIP Allocations feature, but when the new subnets API comes out, you can get rid of all of this complexity. I think that would be a massive win as far as a maintenance burden.

I would keep this validation here for now, but I will let you know if there are any updates.

gcs278 · 2024-10-24T21:07:29Z

/assign

gcs278

Thanks, good information in here. I think validation is the biggest item of contention that we need to solve.

gcs278 · 2024-10-24T21:12:27Z

enhancements/ingress/aws-lb-eip-selection.md

+### Non-Goals
+- Creation of EIPs in AWS.
+- Static IP usage with NLBs for OpenShift API server, DNS, Nat Gateways, LBs, Instances.
+- To assign IPs from a Customer Owned IP (CoIP) Pool when using Outposts.


It's pretty obvious, but maybe worth a mention that this default is not for user-created IngressControllers.

Suggested change

- To assign IPs from a Customer Owned IP (CoIP) Pool when using Outposts.

- To assign IPs from a Customer Owned IP (CoIP) Pool when using Outposts.

- Set default EIPs for user-created IngressControllers

gcs278 · 2024-10-24T21:14:06Z

enhancements/ingress/aws-lb-eip-selection.md

+- To assign IPs from a Customer Owned IP (CoIP) Pool when using Outposts.
+
+## Proposal
+This enhancement adds API fields in the installer and the IngressController specification


I don't think this proposal is adding any APIs to the IngressController right?

Suggested change

This enhancement adds API fields in the installer and the IngressController specification

This enhancement adds API fields in the installer and the Ingress Config specification

gcs278 · 2024-10-24T21:15:51Z

enhancements/ingress/aws-lb-eip-selection.md

+### API Extensions
+
+#### Installer Updates
+- The first API extension for setting `eipAllocations` is in the installer [Platform](https://github.com/openshift/installer/blob/master/pkg/types/aws/platform.go) type, where the new field `NetworkLoadBalancerParameters` is added as an optional field.


I don't see this field name.

Suggested change

- The first API extension for setting `eipAllocations` is in the installer [Platform](https://github.com/openshift/installer/blob/master/pkg/types/aws/platform.go) type, where the new field `NetworkLoadBalancerParameters` is added as an optional field.

- The first API extension for setting `eipAllocations` is in the installer [Platform](https://github.com/openshift/installer/blob/master/pkg/types/aws/platform.go) type, where the new field `eipAllocations` is added as an optional field.

gcs278 · 2024-10-24T21:29:26Z

enhancements/ingress/aws-lb-eip-selection.md

+    // eipAllocations holds eipAllocations for an default AWS
+    // NLB IngressController.


You created this as an struct of EIPAllocations, good idea, so it may be extended in the future if anything else needs EIPs. But shouldn't this go doc reflect that it's a generic structure? Does this reflect your idea of the API?:

Suggested change

// eipAllocations holds eipAllocations for an default AWS

// NLB IngressController.

// eipAllocations contains Elastic IP (EIP) allocations for AWS resources

// within the cluster.

gcs278 · 2024-10-24T21:31:38Z

enhancements/ingress/aws-lb-eip-selection.md

+// EIPAllocations holds configuration parameters for an
+// default AWS NLB IngressController. For Example: Setting AWS EIPs https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html


Also here. I think the link is unnecessary, but keep it if you like it.

Suggested change

// EIPAllocations holds configuration parameters for an

// default AWS NLB IngressController. For Example: Setting AWS EIPs https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html

// EIPAllocations contains Elastic IP (EIP) allocations for AWS resources

// within the cluster.

gcs278 · 2024-10-24T22:12:17Z

enhancements/ingress/aws-lb-eip-selection.md

+
+## Open Questions
+- Q: As per [EP](https://github.com/openshift/enhancements/pull/1634), old subnets field will be deprecated. So, shall we skip the validation for checking
+  number of `BYO Subnets` provided in the `install-config.yaml` with the number of eipAllocations ? Or shall we compare the old subnets field with the eipAllocations ?


I'm 50/50 on doing validation. It's going to be painful to write and maintain, but also it sucks when I install a cluster and realize 45 minutes later that I didn't add enough EIPs. Instant feedback is much better, as long as we get it right.

Like I said in another comment, I'm pushing for a simplification in the new subnets field, where this whole "predicited LB subnet count" goes away. IF that's introduced, validation will be trivial len(subnets) == len(eips). I yield to the installer team on this one.

gcs278 · 2024-10-24T22:19:00Z

enhancements/ingress/aws-lb-eip-selection.md

+## Open Questions
+- Q: As per [EP](https://github.com/openshift/enhancements/pull/1634), old subnets field will be deprecated. So, shall we skip the validation for checking
+  number of `BYO Subnets` provided in the `install-config.yaml` with the number of eipAllocations ? Or shall we compare the old subnets field with the eipAllocations ?
+


Mind adding one more that I'm curious about? CC: @JoelSpeed

Suggested change

- Q: Should we split Ingress Config into defaulting for the default IngressController and defaulting for user-created IngressControllers?

gcs278 · 2024-10-24T22:20:34Z

enhancements/ingress/aws-lb-eip-selection.md

+the Predicted LB Subnets != BYO Subnet Count scenario as not valid? And possibly block future installs as a resolution
+to https://issues.redhat.com/browse/OCPBUGS-17432? That would make EIP Allocation a lot easier, but not sure if that's realistic. 
+
+4. #### Validation to check if EIPs are not already assigned to resources.


What about if the EIP exists? I don't think you explicitly mention that in these sections.

gcs278 · 2024-10-24T22:36:42Z

enhancements/ingress/aws-lb-eip-selection.md

+EIPs can be assigned to many resource types, like Nat Gateways, *LBs, Instances, etc. The attribute associationId will be set when the EIP is already associated.
+To mitigate this we could add that validation, at least on installer, to provide quick-feedback (fail when validate install-config) to the user when the provided EIP is already associated to another resource.
+It would be nice to have a validation before setting the annotation to CCM, keeping the operator degraded before disrupting the service.
+


Could we add a small section on the defaulting mechanics for the IngressController? It's definitely an implementation detail, but an important one that feels important enough to mention here. Something @Miciah or @JoelSpeed might be interested in commenting on. I wonder if doing status "right" was a bit of a mistake, because now things are becoming inconsistent in the API. Either way, that that ship sailed in 4.17.

Suggested change

#### IngressConfig EIP Allocation Defaulting Mechanics for Ingress Controller

Traditionally, the Ingress Operator has populated default values from the Ingress Config into the `status`, making `status` effectively reflect the desired state of the IngressController. However, since `eipAllocations` in `status` represents the **actual** state, not the **desired** state, the default `eipAllocations` values must be set in the `spec` when the Ingress Operator initially admits the IngressController.

This approach is new. The Ingress Operator does not typically set default values in `spec` for load balancer configurations if the user hasn’t explicitly provided them. While this defaulting pattern is more consistent with Kubernetes conventions for `spec` and `status` (and is also our only option in this situation), it's important to acknowledge that this inconsistency in defaulting behavior could cause confusion for users.

gcs278 · 2024-10-24T22:40:07Z

enhancements/ingress/aws-lb-eip-selection.md

+
+### Goals
+- Users are able to use EIPs for a default NLB `IngressController` at install time.
+- Check for unassociated EIPs before passing to CCM.


Think we should make this more generic? Otherwise, you are missing a couple of validations here.

Suggested change

- Check for unassociated EIPs before passing to CCM.

- Add validation to the installer to prevent invalid EIP configurations

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Sep 26, 2024

openshift-ci bot requested review from frobware and rfredette September 26, 2024 03:51

miheer force-pushed the eip-installer branch 3 times, most recently from c79695a to c4312f0 Compare September 26, 2024 04:37

miheer mentioned this pull request Sep 26, 2024

NE-1517: Set EIP for NLB Ingress controller. #1593

Open

miheer force-pushed the eip-installer branch from c4312f0 to 454b5a8 Compare September 26, 2024 05:30

r4f4 reviewed Sep 30, 2024

View reviewed changes

sadasu reviewed Oct 3, 2024

View reviewed changes

miheer force-pushed the eip-installer branch 3 times, most recently from 857c756 to 9f2d0ae Compare October 21, 2024 01:05

CORS-3687: Enhancement proposal for setting EIPs for Ingress Controll…

c39762e

…er via installer

miheer force-pushed the eip-installer branch from 9f2d0ae to c39762e Compare October 21, 2024 06:55

r4f4 reviewed Oct 21, 2024

View reviewed changes

gcs278 reviewed Oct 24, 2024

View reviewed changes

openshift-ci bot assigned gcs278 Oct 24, 2024

gcs278 reviewed Oct 24, 2024

View reviewed changes

	- As a cluster administrator using installer, I want to configure default NLB IngressController to use EIPs.
	- As a cluster administrator using installer, I want to configure `default` NLB IngressController to use EIPs.


		We can examine the following scenarios:

		##### BYO Subnet Count != EIPs Allocations && BYO Subnet Count == Predicted LB Subnet count && Predicted LB Subnet count != EIPs Allocations:

	- To assign IPs from a Customer Owned IP (CoIP) Pool when using Outposts.
	- To assign IPs from a Customer Owned IP (CoIP) Pool when using Outposts.
	- Set default EIPs for user-created IngressControllers

	This enhancement adds API fields in the installer and the IngressController specification
	This enhancement adds API fields in the installer and the Ingress Config specification

	- The first API extension for setting `eipAllocations` is in the installer [Platform](https://github.com/openshift/installer/blob/master/pkg/types/aws/platform.go) type, where the new field `NetworkLoadBalancerParameters` is added as an optional field.
	- The first API extension for setting `eipAllocations` is in the installer [Platform](https://github.com/openshift/installer/blob/master/pkg/types/aws/platform.go) type, where the new field `eipAllocations` is added as an optional field.

		// eipAllocations holds eipAllocations for an default AWS
		// NLB IngressController.

		// EIPAllocations holds configuration parameters for an
		// default AWS NLB IngressController. For Example: Setting AWS EIPs https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html


	- Q: Should we split Ingress Config into defaulting for the default IngressController and defaulting for user-created IngressControllers?

+#### IngressConfig EIP Allocation Defaulting Mechanics for Ingress Controller
+Traditionally, the Ingress Operator has populated default values from the Ingress Config into the `status`, making `status` effectively reflect the desired state of the IngressController. However, since `eipAllocations` in `status` represents the **actual** state, not the **desired** state, the default `eipAllocations` values must be set in the `spec` when the Ingress Operator initially admits the IngressController.
+This approach is new. The Ingress Operator does not typically set default values in `spec` for load balancer configurations if the user hasn’t explicitly provided them. While this defaulting pattern is more consistent with Kubernetes conventions for `spec` and `status` (and is also our only option in this situation), it's important to acknowledge that this inconsistency in defaulting behavior could cause confusion for users.

	- Check for unassociated EIPs before passing to CCM.
	- Add validation to the installer to prevent invalid EIP configurations

CORS-3687: Enhancement proposal for setting EIPs for Ingress Controller via installer #1688

Are you sure you want to change the base?

CORS-3687: Enhancement proposal for setting EIPs for Ingress Controller via installer #1688

Conversation

miheer commented Sep 26, 2024 • edited Loading

openshift-ci-robot commented Sep 26, 2024 • edited by openshift-ci bot Loading

openshift-ci bot commented Sep 26, 2024

miheer commented Sep 26, 2024 • edited Loading

openshift-ci-robot commented Sep 26, 2024 • edited by openshift-ci bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sadasu Oct 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

miheer Oct 21, 2024 • edited Loading

Choose a reason for hiding this comment

miheer commented Oct 21, 2024

openshift-ci bot commented Oct 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gcs278 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gcs278 Oct 24, 2024 • edited Loading

Choose a reason for hiding this comment

gcs278 commented Oct 24, 2024

gcs278 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

miheer commented Sep 26, 2024 •

edited

Loading

openshift-ci-robot commented Sep 26, 2024 •

edited by openshift-ci bot

Loading

miheer commented Sep 26, 2024 •

edited

Loading

openshift-ci-robot commented Sep 26, 2024 •

edited by openshift-ci bot

Loading

sadasu Oct 3, 2024 •

edited

Loading

miheer Oct 21, 2024 •

edited

Loading

gcs278 Oct 24, 2024 •

edited

Loading