stability: iptables analysis

enhancements/kube-apiserver/stability-iptables-analysis.md

@@ -0,0 +1,203 @@
+---
+title: stability-iptables-analysis
+authors:
+  - "@janedoe"
+reviewers:
+  - TBD
+  - "@alicedoe"
+approvers:
+  - TBD
+  - "@oscardoe"
+creation-date: yyyy-mm-dd
+last-updated: yyyy-mm-dd
+status: provisional|implementable|implemented|deferred|rejected|withdrawn|replaced
+see-also:
+  - "/enhancements/this-other-neat-thing.md"  
+replaces:
+  - "/enhancements/that-less-than-great-idea.md"
+superseded-by:
+  - "/enhancements/our-past-effort.md"
+---
+
+# Stability: iptables Analysis
+
+## Release Signoff Checklist
+
+- [ ] Enhancement is `implementable`
+- [ ] Design details are appropriately documented from clear requirements
+- [ ] Test plan is defined
+- [ ] Graduation criteria for dev preview, tech preview, GA
+- [ ] User-facing documentation is created in [openshift-docs](https://github.com/openshift/openshift-docs/)
+
+## Open Questions [optional]
+
+This is where to call out areas of the design that require closure before deciding
+to implement the design.  For instance, 
+ > 1. This requires exposing previously private resources which contain sensitive
+  information.  Can we do this? 
+
+## Summary
+
+In 4.x, one point of fragility has been missing iptables rules.
+There have been at least two potential causes found, but we still don't have an automatic way to figure out how many clusters 
+and services are afflicted.
+Dan Mace found a way to find analyze iptables dumps to see if services and/or endpoints are missing.
+
+## Motivation
+
+If iptables rules are missing, the cluster can behave unpredictably.
+
+### Goals
+
+1. Analyze iptables rules for missing services and endpoints.
+2. Produce a metric and alert for latency in iptables rules on masters.
+
+### Non-Goals
+
+1. Figure out why iptables rules are missing.
+2. Analyze iptables rules on all nodes.  This is too large.
+
+## Proposal
+
+1. Run a daemonset on every master that records iptables rules into a configmap on some reasonable cadence and expiry policy.
+2. Write an analyzer binary that checks to see how far out of date they are. This involves...
+   1. Did the state ever exist?  You can compare resourceversions to determine if the state never existed (in some cases).
+   2. How latent are the iptables?  Based on bookmarking of when endpoints and services were created/updated, 
+      you can determine roughly how latent the iptables rules are.
+3. Write summaries into some resource which you can then report metrics and degraded conditions against.
+
+### User Stories [optional]
+
+Detail the things that people will be able to do if this is implemented.
+Include as much detail as possible so that people can understand the "how" of
+the system. The goal here is to make this feel real for users without getting
+bogged down.
+
+#### Story 1
+
+#### Story 2
+
+### Implementation Details/Notes/Constraints [optional]
+
+What are the caveats to the implementation? What are some important details that
+didn't come across above. Go in to as much detail as necessary here. This might
+be a good place to talk about core concepts and how they relate.
+
+### Risks and Mitigations
+
+What are the risks of this proposal and how do we mitigate. Think broadly. For
+example, consider both security and how this will impact the larger OKD
+ecosystem.
+
+How will security be reviewed and by whom? How will UX be reviewed and by whom?
+
+Consider including folks that also work outside your immediate sub-project.
+
+## Design Details
+
+### Test Plan
+
+**Note:** *Section not required until targeted at a release.*
+
+Consider the following in developing a test plan for this enhancement:
+- Will there be e2e and integration tests, in addition to unit tests?
+- How will it be tested in isolation vs with other components?
+
+No need to outline all of the test cases, just the general strategy. Anything
+that would count as tricky in the implementation and anything particularly
+challenging to test should be called out.
+
+All code is expected to have adequate tests (eventually with coverage
+expectations).
+
+### Graduation Criteria
+
+**Note:** *Section not required until targeted at a release.*
+
+Define graduation milestones.
+
+These may be defined in terms of API maturity, or as something else. Initial proposal
+should keep this high-level with a focus on what signals will be looked at to
+determine graduation.
+
+Consider the following in developing the graduation criteria for this
+enhancement:
+- Maturity levels - `Dev Preview`, `Tech Preview`, `GA`
+- Deprecation
+
+Clearly define what graduation means.
+
+#### Examples
+
+These are generalized examples to consider, in addition to the aforementioned
+[maturity levels][maturity-levels].
+
+##### Dev Preview -> Tech Preview
+
+- Ability to utilize the enhancement end to end
+- End user documentation, relative API stability
+- Sufficient test coverage
+- Gather feedback from users rather than just developers
+
+##### Tech Preview -> GA 
+
+- More testing (upgrade, downgrade, scale)
+- Sufficient time for feedback
+- Available by default
+
+**For non-optional features moving to GA, the graduation criteria must include
+end to end tests.**
+
+##### Removing a deprecated feature
+
+- Announce deprecation and support policy of the existing feature
+- Deprecate the feature
+
+### Upgrade / Downgrade Strategy
+
+If applicable, how will the component be upgraded and downgraded? Make sure this
+is in the test plan.
+
+Consider the following in developing an upgrade/downgrade strategy for this
+enhancement:
+- What changes (in invocations, configurations, API use, etc.) is an existing
+  cluster required to make on upgrade in order to keep previous behavior?
+- What changes (in invocations, configurations, API use, etc.) is an existing
+  cluster required to make on upgrade in order to make use of the enhancement?
+
+### Version Skew Strategy
+
+How will the component handle version skew with other components?
+What are the guarantees? Make sure this is in the test plan.
+
+Consider the following in developing a version skew strategy for this
+enhancement:
+- During an upgrade, we will always have skew among components, how will this impact your work?
+- Does this enhancement involve coordinating behavior in the control plane and
+  in the kubelet? How does an n-2 kubelet without this feature available behave
+  when this feature is used?
+- Will any other components on the node change? For example, changes to CSI, CRI
+  or CNI may require updating that component before the kubelet.
+
+## Implementation History
+
+Major milestones in the life cycle of a proposal should be tracked in `Implementation
+History`.
+
+## Drawbacks
+
+The idea is to find the best form of an argument why this enhancement should _not_ be implemented.
+
+## Alternatives
+
+Similar to the `Drawbacks` section the `Alternatives` section is used to
+highlight and record other possible approaches to delivering the value proposed
+by an enhancement.
+
+## Infrastructure Needed [optional]
+
+Use this section if you need things from the project. Examples include a new
+subproject, repos requested, github details, and/or testing infrastructure.
+
+Listing these here allows the community to get the process for these resources
+started right away.