Merge pull request kubernetes#2 from bridgetkromhout/updates-to-563

Additional updates

keps/sig-network/563-dual-stack/README.md

@@ -1391,22 +1391,27 @@ This capability will move to stable when the following criteria have been met.
  - Components depending on the feature gate: kube-apiserver, kube-controller-manager, kube-proxy, and kubelet
 
 * **Does enabling the feature change any default behavior?**
- No. Pods and Services will remain single stack until cli flags have been modified
+ Pods and Services will remain single-stack until cli flags have been modified
  as described in this KEP. Once modified, existing and new services will remain
- single stack until user requests otherwise. Pods will become dual-stack however
- it will maintain the same ipfamily used in before enabling feature flag.
+ single-stack until user requests otherwise. Pods will become dual-stack
+ however will maintain the same ipfamily used before enabling feature flag.
 
 * **Can the feature be disabled once it has been enabled (i.e. can we roll back
  the enablement)?**
- Yes. If we disable the feature gate, we must remove the CLI parameters. An
- older client won't see or be able to use the new fields.
+ Yes. If you decide to turn off dual-stack after turning on, ensure all
+ services are converted to single-stack first (switch ipfamily to single-stack
+ on all services) and then disable the feature. Remove the CLI parameters, as
+ an older client won't see or be able to use the new fields. When the user
+ disables dual-stack from the controller manager, new endpoints will no longer
+ carry two sets, while existing endpoints may not be updated.
 
 * **What happens if we reenable the feature if it was previously rolled back?**
- Similar to enabling it the first time on a cluster. We don't load balance
- across IP families, and with no selectors we don't get endpoints. If you use
- the feature flag to turn off dual-stack, we do not edit your service. If you
+ Whatever the user has defined will not change without intervention. If you
  disable dual-stack from the controller manager, the service will be given
- single-stack endpoints.
+ single-stack endpoints. If you enable dual-stack again, it's as if you're
+ enabling it for the first time on a cluster. We don't load balance across IP
+ families, and with no selectors we don't get endpoints. If you use the
+ feature flag to turn off dual-stack, we do not edit user input.
 
 * **Are there any tests for feature enablement/disablement?**
  The feature is being tested using integration tests with gate on/off. The
@@ -1420,14 +1425,23 @@ This capability will move to stable when the following criteria have been met.
 ### Rollout, Upgrade and Rollback Planning
 
 * **How can a rollout fail? Can it impact already running workloads?**
- Users **must** avoid changing existing cidrs. For both pods and services. Users
- can only add to alternative ip family to existing cidrs. Changing existing cidrs
- will result in nondeterministic failures depending on how the cluster networking 
- was configured.
-
- Existing workloads are not expected to be impacted during rollout. A component restart
- during rollout might delay generating endpoint and endpoint slices for alternative IP families
- if there are *new* workloads that depend on them they will fail.
+ Users **must** avoid changing existing CIDRs for both pods and services.
+ Users can only add an alternative ip family to existing CIDRs. Changing
+ existing CIDRs will result in nondeterministic failures depending on how the
+ cluster networking was configured.
+
+ Existing workloads are not expected to be impacted during rollout. When you
+ disable dual-stack, existing routes aren't deleted. A component restart
+ during rollout might delay generating endpoint and endpoint slices for
+ alternative IP families. If there are *new* workloads that depend on them,
+ they will fail.
+
+ Imbalance is possible if a replica set scales up or ipv6 gets turned off and
+ the endpoint controller has not yet updated, but iptables and the service
+ controller manager won't look at endpoints that are flagged off. (Endpoints
+ can exist and not be used.) For services, the user will get an error
+ immediately. If the existing rules are removed but new ones don't resolve
+ correctly yet, then it has a chance to resolve on the next loop.
 
 * **What specific metrics should inform a rollback?**
  N/A
@@ -1455,31 +1469,38 @@ fields of API types, flags, etc.?**
 
 * **What are the SLIs (Service Level Indicators) an operator can use to determine 
 the health of the service?**
- Dual-stack networking is a functional addition, not a service with SLIs.
+ Dual-stack networking is a functional addition, not a service with SLIs. Use
+ existing metrics for kubelet pod creation and service creation to determine
+ service health. [Validate
+ IPv4/IPv6 dual-stack](https://kubernetes.io/docs/tasks/network/validate-dual-stack/)
+ to ensure that node addressing, pod addressing, and services are configured
+ correctly. If dual-stack services are created, they have passed validation.
 
 * **What are the reasonable SLOs (Service Level Objectives) for the above SLIs?**
- N/A
+ Existing kubelet pod creation and service creation SLOs are what is needed.
 
 * **Are there any missing metrics that would be useful to have to improve observability 
 of this feature?**
 
  1. For services:
 
- Services are not in the path of pod creation. Thus, any malfunction or
- bugs in services will not affect pods.
+ Whether a cluster is converted to dual-stack or converted back to
+ single-stack, services will remain the same because the dual-stack
+ conversation does not change user data.
 
  Services/Endpoint selection is not in path of pod creation. It runs in
- kube-controller-manager, thus this is N/A.
+ kube-controller-manager, so any malfunction will not affect pods.
 
  2. For pods:
 
  Dual-stack components are not in path of pod creation. It is in the path
  of reporting pod ips. So pod creation will not be affected; if it is
- affected, then it is a CNI issue.
+ affected, then it is a CNI issue (entirely separate from dual-stack).
 
  Dual-stack components are in the path of PodIPs reporting which affects
  kubelet. If there is a problem (or if there are persistent problems)
- then disabling the featuregate on api-server will mitigate.
+ then it is migitated by disabling the feature gate, which turns it off
+ for kube-apiserver, kube-controller-manager, kube-proxy, and kubelet.
 
 ### Dependencies
 
@@ -1497,8 +1518,7 @@ of this feature?**
 
 * **Will enabling / using this feature result in any new calls to the cloud 
 provider?**
- No. Errors are surfaced when the user makes a call. We don't focus on
- metrics-server at this point.
+ No. IP allocation for services only involves the API server.
 
 * **Will enabling / using this feature result in increasing size or count of 
 the existing API objects?**
@@ -1524,7 +1544,7 @@ resource usage (CPU, RAM, disk, IO, ...) in any components?**
  * Failure to create dual-stack services. Operator must perform the following steps:
  1. Ensure that the cluster has `IPv6DualStack` feature enabled.
  2. Ensure that api-server is correctly configured with multi (dual-stack) service
- cidrs using `--services-cluster-ip-range` flag.
+ CIDRs using `--services-cluster-ip-range` flag.
 
  * Failure to route traffic to pod backing a dual-stack service. Operator must perform the 
  following steps:
@@ -1543,6 +1563,22 @@ resource usage (CPU, RAM, disk, IO, ...) in any components?**
  5. If the pod is using host network then operator must ensure that the node is correctly
  reporting dual-stack addresses.
 
+ * Possible workload imbalances
+
+ Once a service is saved (with the appropriate feature flag and one or more
+ cidrs), the controller manager scans the number of pods (seeing dual or
+ single) and creates endpoint objects pointing to these pods.Each node has
+ a proxy that watches for services, endpoints, endpoint slices, per family.
+ If the existing rules are removed but new ones don't resolve correctly, then
+ it will resolve on the next loop.
+
+ If allocation fails, that would translate to the pod not running if there is
+ no IP allocated from the CIDR. If the pod is created but routing is not
+ working correctly, there will be an error in the kube-proxy logs, so
+ debugging would take place by looking at iptables, similar to how it is
+ already done today. The cluster IP allocation is allocated in a syncronous
+ process; if this fails the service creation will fail and the service object
+ will not be persisted.
+
 * **What steps should be taken if SLOs are not being met to determine the problem?**
  N/A
-