Skip to content

Commit 83bd957

Browse files
mtuliomtulio
committed
review goal statements
1 parent e15ba48 commit 83bd957

File tree

1 file changed

+76
-188
lines changed

1 file changed

+76
-188
lines changed

‎enhancements/installer/aws-ingress-nlb-security-group.md‎

Lines changed: 76 additions & 188 deletions
Original file line numberDiff line numberDiff line change
@@ -78,251 +78,139 @@ Topologies/deployment:
7878

7979
#### Option 1. Opt-in NLB provisioning with Security Groups for Default Ingress with NO SG control by CCM
8080

81-
Users will be able to deploy OpenShift on AWS with the default Ingress using Network Load Balancers with Security Groups when enabled (opt-in) in the install-config.yaml. Installation agent (openshift-install, hypershift) manages the Security Group creation.
81+
Users will be able to deploy OpenShift on AWS with the default Ingress using Network Load Balancers with Security Groups when enabled (opt-in) in the `install-config.yaml`. The installation agent (`openshift-install`, `hypershift`) manages the Security Group creation.
8282

83-
Hightlights:
84-
- Focus on short-term resolving security issues when not using SG on NLB
85-
- Focus on customer scalation of enabling SG on NLB
86-
- Minimum changes on CCM
83+
Highlights:
84+
- Focus on short-term resolution of security issues when not using SG on NLB.
85+
- Focus on customer scalability when enabling SG on NLB.
86+
- Minimal changes to CCM.
8787

8888
T-Shirt Sizing/complexity by component:
89-
| Component | T-Shirt Sz | Complexity | Note |
89+
| Component | T-Shirt Size | Complexity | Note |
9090
| -- | -- | -- | -- |
91-
| CCM | S | S | No API changes, No SG management, Opt-in |
92-
| CIO | S | S | API adding SG ID/Name to service Annotation |
93-
| Installer | S | S | API enabling feature; Creates Ingress SG (SDK) |
94-
| ROSA CL | M? | M? | TBD:API enabling feature(?); creates Ingress SG; update install-config |
95-
| ROSA HCP | M? | M? | TBD:API enabling feature(?); SG mgr; create CIO manifests to enable SG |
96-
| Day-2 | S | M | BYO SG (can managed svc automate through cli?), patch CIO to recreate NLB; |
97-
91+
| CCM | S | S | No API changes, No SG management, Opt-in. |
92+
| CIO | S | S | API adds SG ID/Name to service annotation. |
93+
| Installer | S | S | API enabling feature; Creates Ingress SG (SDK). |
94+
| ROSA CL | M? | M? | TBD: API enabling feature(?); creates Ingress SG; updates `install-config`. |
95+
| ROSA HCP | M? | M? | TBD: API enabling feature(?); SG mgt; creates CIO manifests to enable SG. |
96+
| Day-2 | S | M | BYO SG (can managed services automate through CLI?), patch CIO to recreate NLB. |
9897

9998
Risk:
100-
- Upstream CCM changes can take longer than expected (Small change can carry dowstream)
101-
- Default Ingress changes the port from 80 and 443(happen??) requires manual SG's rules updates
99+
- Upstream CCM changes can take longer than expected (small changes may propagate downstream).
100+
- Default Ingress changes to ports 80 and 443 (if they occur) require manual SG rule updates.
102101

103102
Day-2 update:
104-
- Self-manage: require user to create SG and patch CIO
105-
- Manage: require to create SG and patch CIO (can be automated by CLI)
106-
- Updates trigger service recreating NLB
103+
- Self-managed: Requires the user to create SG and patch CIO.
104+
- Managed: Requires creating SG and patching CIO (can be automated via CLI).
105+
- Updates trigger service recreation of NLB.
107106

108107
**Phase 1: Create support on Self-Managed**
109108

110109
Goals:
111-
- Installer manifest stage: when SG is enabled, set the security group names to the CIO manifest
112-
- Installer infra stage: when SG is enabled, create security group and rules for ingress (InfraReady hook)
110+
- Installer manifest stage: When SG is enabled, set the security group names in the CIO manifest.
111+
- Installer infra stage: When SG is enabled, create the security group and rules for ingress (InfraReady hook).
113112
- Cluster-Ingress-Operator:
114-
- API to receipt the Security Group in the NLB parameters
115-
- Service Controller create the annotation `service.beta.kubernetes.io/aws-load-balancer-security-groups` with custom Security Group
116-
- Enable support of BYO (unmanaged) Security Groups for AWS Cloud Controller Manager (CCM/cloud-provider-aws)
117-
113+
- API to receive the Security Group in the NLB parameters.
114+
- Service Controller creates the annotation `service.beta.kubernetes.io/aws-load-balancer-security-groups` with the custom Security Group.
115+
- Enable support for BYO (unmanaged) Security Groups for AWS Cloud Controller Manager (CCM/cloud-provider-aws).
118116

119117
**Phase 2: Create support on ROSA Classic**
120118

121119
Goals:
122-
- TBD how ROSA can enable the option on install-config: `platform.aws.ingressController.SecurityGroupEnabled`
120+
- TBD: How ROSA can enable the option in `install-config`: `platform.aws.ingressController.SecurityGroupEnabled`.
123121

124122
**Phase 3: Create support on ROSA HCP**
125123

126124
Goals:
127-
- TBD hypershift creating ingress Security Group for the ingress (after VPC creation)
128-
- TBD hypershift setting the Security Group in CIO manifests
125+
- TBD: Hypershift creates the ingress Security Group for the ingress (after VPC creation).
126+
- TBD: Hypershift sets the Security Group in CIO manifests.
129127

128+
---
130129

131130
#### Option 2. Opt-in NLB provisioning with Security Groups for Default Ingress with FULL SG control by CCM
132131

133-
Users will be able to deploy OpenShift on AWS with the default Ingress using Network Load Balancers with Security Groups when enabled (opt-in) in the install-config.yaml. CCM-AWS full manages the SG lifecycle (similar CLB, but as opt-in through annotation).
134-
135-
Hightlights:
136-
- Focus on short-term resolving security issues when not using SG on NLB
137-
- Focus on customer scalation of enabling SG on NLB
138-
- Medium changes on CCM
132+
Users will be able to deploy OpenShift on AWS with the default Ingress using Network Load Balancers with Security Groups when enabled (opt-in) in the `install-config.yaml`. CCM-AWS fully manages the SG lifecycle (similar to CLB, but as opt-in through annotation).
139133

134+
Highlights:
135+
- Focus on short-term resolution of security issues when not using SG on NLB.
136+
- Focus on customer scalability when enabling SG on NLB.
137+
- Moderate changes to CCM.
140138

141139
T-Shirt Sizing/complexity by component:
142-
143-
| Component | T-Shirt Sz | Complexity | Note |
140+
| Component | T-Shirt Size | Complexity | Note |
144141
| -- | -- | -- | -- |
145-
| CCM | M | M | API introducing annotation to "create SG on NLB"(default for CLB) |
146-
| CIO | S | S | API adding SG ID/Name to service Annotation |
147-
| Installer | S | S | no SG mgr; API enabling feature; |
148-
| ROSA CL | S? | S? | no SG mgr; update install-config |
149-
| ROSA HCP | S? | S? | no SG mgr; create CIO manifests to "enable NLB with SG" |
150-
| Day-2 | S | S | patch CIO to recreate NLB |
151-
142+
| CCM | M | M | API introduces annotation to "create SG on NLB" (default for CLB). |
143+
| CIO | S | S | API adds SG ID/Name to service annotation. |
144+
| Installer | S | S | No SG mgt; API enabling feature. |
145+
| ROSA CL | S? | S? | No SG mgt; updates `install-config`. |
146+
| ROSA HCP | S? | S? | No SG mgt; creates CIO manifests to "enable NLB with SG". |
147+
| Day-2 | S | S | Patch CIO to recreate NLB. |
152148

153149
Risk:
154150
- CCM/upstream:
155151
- SG management increases controller complexity and scenarios to validate.
156-
- API changes
157-
- More changes in upstream increases the consensus/approvals, specially new features in service LB on CCM (prefer ALBC)
158-
- More cchanges in CCM making it creates and manage SG lifecycle
152+
- API changes.
153+
- More changes upstream increase the consensus/approvals required, especially for new features in service LB on CCM (prefer ALBC).
154+
- More changes in CCM to create and manage the SG lifecycle.
159155
- CCM/downstream:
160-
- more complex to carry when not in upstream
156+
- More complex to maintain when not upstreamed.
161157

162158
Day-2 update:
163-
- Self-managed: patch CIO
164-
- Managed Services: patch CIO
165-
- Updates trigger service recreating NLB
159+
- Self-managed: Patch CIO.
160+
- Managed Services: Patch CIO.
161+
- Updates trigger service recreation of NLB.
166162

163+
---
167164

168165
#### Option 3. NLB feature parity on CCM with ALBC
169166

170167
T-Shirt Sizing/complexity by component:
171-
172-
| Component | T-Shirt Sz | Complexity | Note |
168+
| Component | T-Shirt Size | Complexity | Note |
173169
| -- | -- | -- | -- |
174-
| CCM | XXL | XXL | NLB Feature parity plan with ALBC. Long-term commitment and support by RH |
175-
| CIO | S | S | API adding SG ID/Name to service Annotation |
176-
| Installer | S | S | no SG mgr; API enabling feature; |
177-
| ROSA CL | S? | S? | no SG mgr; update install-config |
178-
| ROSA HCP | S? | S? | no SG mgr; create CIO manifests to "enable NLB with SG" |
179-
| Day-2 | S | S | patch CIO to recreate NLB |
170+
| CCM | XXL | XXL | NLB feature parity plan with ALBC. Long-term commitment and support by RH. |
171+
| CIO | S | S | API adds SG ID/Name to service annotation. |
172+
| Installer | S | S | No SG mgt; API enabling feature. |
173+
| ROSA CL | S? | S? | No SG mgt; updates `install-config`. |
174+
| ROSA HCP | S? | S? | No SG mgt; creates CIO manifests to "enable NLB with SG". |
175+
| Day-2 | S | S | Patch CIO to recreate NLB. |
180176

177+
---
181178

182179
#### Option 4. CIO switches to ALBC
183180

184181
T-Shirt Sizing/complexity by component:
185-
186-
| Component | T-Shirt Sz | Complexity | Note |
182+
| Component | T-Shirt Size | Complexity | Note |
187183
| -- | -- | -- | -- |
188-
| CCM | - | - | CCM will not be used by default router |
189-
| CIO | XXL | XL | API: (short-term) opt-in NLB provisioning with SG using ALBC; long-term: all new provisioning with ALBC; move imgs to payload; manage oper lifecycle(perms, etc) |
190-
| Installer | S | S | no SG mgr; API enabling feature; |
191-
| ROSA CL | S? | S? | no SG mgr; update install-config |
192-
| ROSA HCP | S? | S? | no SG mgr; create CIO manifests to "enable NLB with SG" |
193-
| Day-2 | S | S | patch CIO to recreate NLB |
184+
| CCM | - | - | CCM will not be used by the default router. |
185+
| CIO | XXL | XL | API: (short-term) opt-in NLB provisioning with SG using ALBC; (long-term) all new provisioning with ALBC; move images to payload; manage operator lifecycle (permissions, etc.). |
186+
| Installer | S | S | No SG mgt; API enabling feature. |
187+
| ROSA CL | S? | S? | No SG mgt; updates `install-config`. |
188+
| ROSA HCP | S? | S? | No SG mgt; creates CIO manifests to "enable NLB with SG". |
189+
| Day-2 | S | S | Patch CIO to recreate NLB. |
194190

191+
---
195192

196193
#### Option 1+4. Short-term security fixes with long-term Ingress improvements
197194

198-
> Note: can be 2+4 too, but will impact in the resource investiments in feature parity with CCM.
195+
> Note: Can be 2+4 too, but this will impact resource investments in feature parity with CCM.
199196
200-
In short-term support NLB provisioning with Security Group as opt-in by CCM, resolving security issues,
201-
and get long-term modernization by inheriting features added in upstream project ALBC which is actively maitained by AWS.
202-
203-
Hightlights:
204-
- Resolves on short-term resolving security issues when not using SG on NLB
205-
- Resolves customer scalation of enabling SG on NLB
206-
- Medium changes on CCM
207-
- Provide long-term plan to modernize Load Balancer features by inheriting updates from controller on upstream project maitained by Community (and AWS)
197+
In the short term, support NLB provisioning with Security Groups as opt-in by CCM, resolving security issues, and achieve long-term modernization by inheriting features added in the upstream ALBC project, which is actively maintained by AWS.
208198

199+
Highlights:
200+
- Resolves short-term security issues when not using SG on NLB.
201+
- Improves customer scalability when enabling SG on NLB.
202+
- Moderate changes to CCM.
203+
- Provides a long-term plan to modernize Load Balancer features by inheriting updates from the controller in the upstream project maintained by the community (and AWS).
209204

210205
T-Shirt Sizing/complexity by component:
211-
212-
| Component | T-Shirt Sz | Complexity | Note |
206+
| Component | T-Shirt Size | Complexity | Note |
213207
| -- | -- | -- | -- |
214-
| CCM | S | S | No API changes, No SG management, Opt-in |
215-
| CIO | XXL | XL | API: (short-term): opt-in NLB provisioning with SG using ALBC; (long-term): all new provisioning with ALBC; move imgs to payload; manage oper lifecycle(perms, etc) |
216-
| Installer | S | S | API enabling feature; Creates Ingress SG (SDK) |
217-
| ROSA CL | M? | M? | TBD:(short): API enabling feature(?); creates Ingress SG; update install-config; (long):fix many issues |
218-
| ROSA HCP | M? | M? | TBD: (short): API enabling feature(?); SG mgr; create CIO manifests to enable SG; (long):fix many issues | |
219-
| Day-2 | S | M | (short): BYO SG (can managed automate through cli?), (short&long): patch CIO to recreate NLB; |
220-
221-
222-
223-
<!--
224-
===> DRAFT/old notes:
225-
226-
#### Option 1. CCM configures the SG when deploying NLB. (Minimum support)
227-
228-
CCM enable SG when provisioning NLB only when the annotation XXX with SG ID is added.
229-
230-
Pros:
231-
- Minor changes in CCM (upstream): BYO SG approach (similar [LBC annotation `alb.ingress.kubernetes.io/security-groups`](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/ingress/annotations/#security-groups))
232-
- SG is not created by CCM, and CCM is not adding rules (preserving LBC behavior for annotations security-groups and manage-backend-security-group-rules)
233-
- Does not required API updates for CCM (right, @joel?)
234-
- We can stick only the default ingress/router using that approach, requiring any new ingress to use LBC-ish (already supported)
235-
236-
Cons:
237-
- Installer (or CIO) needs to create the SG, passing to the CCM with required rules
238-
- BYO SG approach for CCM would require CIO managing SG rules when provisioning new ingress
239-
- or at least validate if exists
240-
- or not support this feature for other services than the default router, leading user to use ALBO/LBC
241-
242-
243-
PROBLEM:
244-
- installer can't create SG in the manifest phase, so it wont have IDs, only names (candidate)
245-
246-
#### Option 2. CCM manages the SG when deploying NLB. (may require API changes and more complexity)
247-
248-
Pros:
249-
- Installer (or CIO) does not required to manage the SG
250-
- Increase UX as users don't need to handle with SG in Day-2 when creating new ingress
251-
252-
Cons:
253-
254-
- Requires more complexity to manage SG when NLB on CCM (upstream). SG management logic already exists in the CLB controller
255-
- May require API add/change to determine when SG will be enabled for NLB. Example of LBC annotation manage-backend-security-group-rules
256-
257-
258-
#### Option 3. Use AWS LBC to deploy Ingress NLB. (more complex downstream, no changes upstream)
259-
260-
Changes:
261-
- Requires installing ALBO from OLM during installation.
262-
263-
Pros:
264-
- Unlocks new NLB features added to LBC for OpenShift ingress.
265-
266-
Cons:
267-
- Defaulting to LBC may introduce unknown issues.
268-
- Be careful.
269-
- Create a clear migration path.
270-
271-
Open questions:
272-
273-
- Is there an advantage to using LBC as the default service type LB controller (disabling CCM service-LB capabilities)?
274-
- LBC does not support CLBs.
275-
- Pros: Improves UX as default service LBs will use the latest controller features for LB.
276-
- Cons: Breaking change for apps using default service config (with CLB).
277-
- AWS has advised against using CLBs for years. This will prepare OCP to follow cloud-provider best practices. It's important to have a clear deprecation path.
278-
279-
#### Option 1+3. Focus on "enabling" SG on NLB using BYO SG on CCM, and plan the migration to LBC
280-
281-
Laser focus on short-term customer issues and unlock OpenShift's long-term ingress features with NLB.
282-
283-
Pros:
284-
- Delivers customer requirements ASAP.
285-
- Reduces changes in upstream CCM.
286-
- Enables OpenShift to consume features available on LBC.
287-
- Prevents duplicate maintenance of CCM/LBC codebases.
288-
289-
Cons:
290-
- ?
291-
292-
Phases for #1+3:
293-
294-
- Phase 1 (TP SGs for NLB using CCM BYO SG):
295-
- Patch CCM with required changes for BYO SG when using NLB.
296-
- Create APIs for BYO SG on CIO (NLB only).
297-
- Installer:
298-
- Add a configuration option in `install-config` to "use SG" when `lbType == NLB`.
299-
- Provision SG with expected rules (also adding those to nodes' SG).
300-
301-
- Phase 2 (GA SGs for NLB, Dev Preview LBC through CIO):
302-
- Enable CIO provisioning using LBC/ALBO when using NLB.
303-
- CIO API to enable LBC service provisioning (e.g., manage-backend-security-group-rules).
304-
- Retain API added in Phase 1 for future BYO SG use (user-facing, exposed in `install-config`).
305-
- Installer:
306-
- Add a configuration option in `install-config` to opt-in ingress provisioning with LBC. (or make the flag "Use SG when NLB" to trigger the LBC)
307-
- When enabled, do not provision SGs by installer (delegate to CIO->LBC).
308-
- Add a validation webhook to inform users about Classic Load Balancer (CLB) desconuation(recommendation to use NLB) when creating type LB services (non-NLB).
309-
- Alternatively, provide an informational message advising users to use ingress provisioned with CIO, which uses NLBs, to follow best practices and prevent future deprecations.
310-
- Existing NLBs managed by CCM:
311-
- How to make controllers aware of this? Can we live with two controllers, making the controllers to know that the NLB has been provisioned by each one.
312-
313-
- Phase 3 (TP LBC on CIO, Dev Preview NLB by default CIO):):
314-
- Installer:
315-
- Default to provisioning CIO services with LBC (defaults to lbType==NLB <- dev preview this?).
316-
- Customers can use CLB as opt-in
317-
318-
- Phase 4 (GA LBC on CIO, TP NLB by default CIO):
319-
- Installer:
320-
- Default to provisioning CIO services with LBC (defaults to lbType==NLB <- TP this?).
321-
322-
- Phase 5: (GA )
323-
- Make AWS LBC the default controller for service type LoadBalancer.
324-
- Classic Load Balancer is no longer available for CIO (only when customers wants to create service-LB directly)
325-
-->
208+
| CCM | S | S | No API changes, No SG management, Opt-in. |
209+
| CIO | XXL | XL | API: (short-term) opt-in NLB provisioning with SG using ALBC; (long-term) all new provisioning with ALBC; move images to payload; manage operator lifecycle (permissions, etc.). |
210+
| Installer | S | S | API enabling feature; Creates Ingress SG (SDK). |
211+
| ROSA CL | M? | M? | TBD: (short-term) API enabling feature(?); creates Ingress SG; updates `install-config`; (long-term) fixes many issues. |
212+
| ROSA HCP | M? | M? | TBD: (short-term) API enabling feature(?); SG mgt; creates CIO manifests to enable SG; (long-term) fixes many issues. |
213+
| Day-2 | S | M | (short-term) BYO SG (can managed services automate through CLI?), (short-term & long-term) patch CIO to recreate NLB. |
326214

327215
### Non-Goals
328216

0 commit comments

Comments
 (0)