add SR-IOV GA proposal

enhancements/20190903-SRIOV-GA.md

@@ -0,0 +1,178 @@
+title: SR-IOV GA
+authors:
+  - "@zshi-redhat"
+  - "@pliurh"
+  - "@s1061123"
+reviewers:
+  - "@fepan"
+  - "@dougbtv"
+  - "@vpickard"
+  - "@dcbw"
+  - "@squeed"
+approvers:
+  - "@derekwaynecarr"
+creation-date: 2019-09-03
+last-updated: 2019-09-03
+status: implementable
+---
+
+# Graduate SR-IOV to GA
+
+## Release Signoff Checklist
+
+- [x] Enhancement is `implementable`
+- [ ] Design details are appropriately documented from clear requirements
+- [ ] Test plan is defined
+- [ ] Graduation criteria for dev preview, tech preview, GA
+- [ ] User-facing documentation is created in [openshift/docs]
+
+## Summary
+
+A proposal to graduate SR-IOV support from Tech Preview to GA.
+
+SR-IOV networks are enabled by Multus on secondary-onward pod interfaces.
+It is supported as Tech Preview on OpenShift 4.1 and 4.2 releases.
+This proposal is to make SR-IOV Generally Available in 4.3.
+
+## Motivation
+
+* SR-IOV is currently Technical Preview, but customers need to build supported
+  solutions on the technology.
+
+### Goals
+
+- Enhance Multus to support runtimeConfig for IP assignment
+- Enhance Multus to handle default route overwrite
+- Add admission controller for SR-IOV Network Operator
+- Compare SR-IOV network performance w/wo NUMA affinity
+- Support SR-IOV QoS flags
+- Add SR-IOV Metrics and Alerting
+- Support both RHEL and RHCOS worker nodes
+
+### Non-Goals
+
+- GA userspace (dpdk) workload support
+
+## Proposal
+
+### Implementation Details
+
+#### Multus
+
+Static IPAM plugin is used to assign IPv4 and IPv6 addresses statically to
+container interface. It supports receiveing IPv4/v6 address configuration from
+both CNI config and runtimeConfig. However, Multus only supports passing IP
+configuration via CNI config. This proposal is to add runtimeConfig support
+in Multus CNI.
+
+##### Default Route Overwrite
+
+Every pod comes with a default route added via openshift-sdn. It is not
+possible to change this default route to secondary-onward interfaces.
+This proposal is to come up with a solution for Multus to handle default
+route overwriting.
+
+##### DHCP Configuration Guide
+
+DHCP IPAM plugin configuration is complicated due to its architecture,
+for example, DHCP IPAM requires 'DHCP CNI daemon' and 'DHCP server'.
+Currently we only provide simple document and there were several inquiries
+from user to enhance this document. This proposal is to provide DHCP
+configuration guide to help user understand DHCP CNI configuration and
+troubleshooting.
+
+##### Mac Address Pool Management
+
+Fixed MAC address can be specified via pod annotation for SR-IOV devices.
+However, user has to manually add MAC address for each SR-IOV interface
+in pod annotation which is error prone and inefficient. This proposal is
+to improve user experience by providing a MAC pool management tool which
+automatically assigns a MAC address in pod annotation.
+
+#### SR-IOV Operator
+
+Currently, the Custom Resources of SR-IOV Network Operator can only be
+validated by the declarative OpenAPI v3 schema validation rules defined
+in CRD and setting default value for CR is not supported. 
+
+To improve the user experience, an admission controller webhook for the
+CRD of SR-IOV network Operator shall be implemented. The admission controller
+webhook is expected to validate and set default value for the Custom Resource.
+
+The admission controller webhook and its RBAC shall be provisioned to
+openshift master nodes as a deployment by the SRIOV network operator.
+
+The admission controller webhook is expected to:
+- Set default value for fields of SriovNetwork CR, if not specified by user
+- Set default value for fields of SriovNetworkNodePolicy CR, if not specified
+  by user
+- Validate during SriovNetwork CR creation and updating request. The webhook
+  shall be able to validate following errors:
+  - `networkNamespace` refer to non-existing namespace
+- Validate during SriovNetworkNodePolicy CR creation and updating request
+  The webhook shall be able to validate following errors:
+  - The `numVFs` is larger than the maximum value that the NIC can support
+  - The `nicSelector` selects PFs which has already been covered by existing
+    Policy CRs
+  - The `resourceName` contains invalid characters
+  - `pfName` or `rootDevices` refers to non-existing device.
+
+
+#### SR-IOV QoS Flags
+
+Enable VF flags for security and QoS features.
+
+- Spoof Check
+- Trusted VF
+- Max rate limit
+- Min rate limit
+
+### Risks and Mitigations
+
+#### NUMA
+
+NUMA Awareness (aka Topology Manager) will not be available when SR-IOV GA.
+SR-IOV VF devices requested by pod will not be guaranteed to have NUMA
+locality preferences. Topology Manager will likely be included in 4.4 or 4.5
+which has a dependency on upstream development.
+
+## Design Details
+
+### Test Plan
+
+Build baremetal CI environment for the following tests:
+
+- SR-IOV Operator e2e test
+- SR-IOV CNI, Device Plugin and Admission Controller e2e test
+- Upgrade, negative and performance test
+
+### Graduation Criteria
+
+From Tech Preview to GA
+
+#### Tech Preview -> GA
+
+- Manage SR-IOV end to end deployment and configuration via Operator
+- Sufficient test coverage (upgrade, negative and performance)
+- IP and MAC address management for SR-IOV interfaces
+- Security and QoS support for SR-IOV interfaces
+
+### Upgrade / Downgrade Strategy
+
+### Version Skew Strategy
+
+## Implementation History
+
+### Version 4.1
+
+Tech Preview support for SR-IOV CNI and Network Device Plugin
+
+### Version 4.2
+
+Tech Preview support for SR-IOV Operator, CNI, Network Device Plugin and
+Admission Controller
+
+### Version 4.3
+
+GA support for SR-IOV Operator, CNI, Network Device Plugin and Admission
+Controllers