Merge pull request #850 from mariomac/enable-flows-collection

Enable flows collection

enhancements/network/ovs-flow-export-configuration.md

@@ -0,0 +1,267 @@
+---
+title: ovs-flow-export-configuration
+authors:
+  - "@mariomac"
+reviewers:
+  - "@astoycos"
+  - "@jotak"
+approvers:
+  - TBD
+creation-date: 2021-07-29
+last-updated: 2021-07-29
+status: implementable
+see-also:
+  - "/enhancements/network/netflow.md"
+replaces:
+superseded-by:
+---
+
+# OVS flow export configuration
+
+## Release Signoff Checklist
+
+- [x] Enhancement is `implementable`
+- [X] Design details are appropriately documented from clear requirements
+- [X] Test plan is defined
+- [ ] Operational readiness criteria is defined
+- [ ] Graduation criteria for dev preview, tech preview, GA
+- [ ] User-facing documentation is created in [openshift-docs](https://github.com/openshift/openshift-docs/)
+
+## Summary
+
+This document describes configuration changes and requirements at the CNO/OVN level
+to enable network observability as depicted in the high level
+[Enhancement proposal for Network Observability](https://github.com/openshift/enhancements/pull/921).
+
+It can be seen as a follow-up of a previous enhancement
+[bringing Network Flows export support](./netflow.md) to OpenShift when used with
+OVNKubernetes. In this follow-up, advanced configuration for OvS flows export should
+be made possible, and a mechanism must be set up to allow interactions between the
+Cluster Network Operator (CNO) and the Network Observability Operator (NOO - development
+in progress).
+
+## Motivation
+
+The current flow export feature in the CNO has a couple of limitations that
+can affect performance, scalability and user experience in the context of
+the new NOO (Network Observability Operator):
+
+* Performance, because flows (IPFIX) emission, collection and processing may
+have an overall impact in the system performance, that can be mitigated with
+OvS settings such as sampling rate or cache configuration which aren't currently
+exposed. Thus, it is important to expose them.
+
+* Scalability, because the OvS configuration is currently shared by all nodes
+in the cluster, ie. they are all sharing the same IP for pushing to a collector.
+While this might not necessarily be an issue, we want to offer the users the
+ability to deploy the collector as a DaemonSet in order to reduce the in-cluster
+traffic overhead generated by the flow exports, hence having each OvS pushing
+to different IPs (their host IP).
+
+* User experience, because we want the NOO to be as simple as possible to configure,
+deploy and use. To avoid having the users configuring both the CNO and the NOO
+for flow exports, we need a mechanism to allow the two operators to cooperate and
+share some information, without conflicting in the OvS configuration.
+
+### Goals
+
+The high level goal is to bring all necessary changes in the CNO and OVN so that
+the new NOO can pursue its mission. More in details, it means:
+
+* Expose more configuration options for OvS flow exports through OVN and the CNO.
+* Create a mechanism to allow on-host communication between OvS and the flow collector.
+* Create a mechanism to allow information sharing between the CNO and the NOO.
+
+### Non-Goals
+
+To discuss the architecture of the flow collection and processing pipeline. This
+has been already discussed and defined by the Network Observability Team
+by another channel and will potentially lead to other enhancement proposals later.
+
+This document does not define the work required to configure and deploy the
+flow collection and storage pipeline.
+
+It does not define the design and the goal of the Network Observability Operator, or any interaction between the Console or the CLI and NOO in order to enable flow collection.
+
+## Proposal
+
+### User Stories
+
+#### Tuning flows' emission parameters
+
+Some setups might lead to an overcongestion of the cluster resources such as the network or the
+flows' ingestion/processing/storage pipeline (e.g. because the size and frequency of reported
+flows is too high).
+
+To minimize it, the user should be able to tune up the flows' cache and sampling mechanism in order to control
+the number of records sent to collector.
+
+According to the [OpenVSwitch configuration options](https://www.openvswitch.org/support/dist-docs/ovs-vswitchd.conf.db.5.html),
+there are multiple configuration options that could be exposed to the user:
+
+* `cache_active_timeout`: optional integer, in range 0 to 4,200
+  The maximum period in seconds for which an IPFIX flow record  is
+  cached  and  aggregated  before  being  sent.  If not specified,
+  defaults to 0. If 0, caching is disabled.
+
+* `cache_max_flows`: optional integer, in range 0 to 4,294,967,295
+  The maximum number of IPFIX flow records that can be cached at a
+  time.  If  not  specified,  defaults to 0. If 0, caching is disabled.
+
+* `sampling`: optional integer, in range 1 to 4,294,967,295
+  The  rate  at  which  packets should be sampled and sent to each
+  target collector. If not specified, defaults to 400, which means
+  one  out of 400 packets, on average, will be sent to each target
+  collector.
+
+Other arguments might be considered in the future, such as `enable-input-sampling`, `enable-output-sampling`,
+or `enable-tunnel-sampling`.
+
+
+### Implementation Details/Notes/Constraints [optional]
+
+In order to ensure an optimal usage of the network, we allow restricting flow's
+emission-collection traffic to the local node.
+
+The collector can be deployed as a DaemonSet, to ensure that there will be an
+instance of the collector for each node.
+
+Exposing the collector endpoint as a Kubernetes ClusterIP Service would cause the traffic
+to be distributed across all the nodes (there is a `ServiceInternalTrafficPolicy`
+cluster-wide feature gate but, at the moment of writing this document, it is not implemented in
+OVN-Kubernetes, and it's beta in Kubernetes 1.22).
+
+To limit the flows' traffic to internal nodes, each collector should expose a node
+port, so each OVN instance will discover the Node IP (instead of the Service IP) and submit
+data to the NodeIP:NodePort address. The Node IP can be discovered.
+
+From the user-side configuration, the NOO has a custom resource definition that covers IPFix cache and sampling settings, deployment model (as a DaemonSet or as Deployment+Service) and a few other settings not relevant here.
+
+The NOO will generate a `ConfigMap` that is available for reading / watching to the CNO. The CNO would configure the OVN pods with the `ConfigMap` data.
+
+This solution ensures the NOO never interacts directly with OvS in order to avoid any configuration or state conflict with the CNO, and avoids creating a hard dependency in the CNO towards the NOO.
+
+Given the aforementioned details, following subsections depicts the work to do,
+grouped in different tasks.
+
+#### Task 1: setup a new ConfigMap that is readable by the CNO
+
+[Current `Network` operator configuration allows setting a static host:port address](
+https://docs.okd.io/latest/networking/ovn_kubernetes_network_provider/tracking-network-flows.html#nw-network-flows-object_tracking-network-flows).
+This information is forwarded to each OVN instance, on each cluster node. For example:
+
+```yaml
+apiVersion: operator.openshift.io/v1
+kind: Network
+metadata:
+  name: cluster
+spec:
+  exportNetworkFlows:
+    collectors:
+      - 192.168.1.99:2056
+```
+
+However, we need to set this address dynamically to ensure that each pod will forward
+the flows to the collector in its same node.
+
+While the above `Network` configuration is still valid for the current use cases, the NOO will generate a new `ConfigMap` that is exposed to the CNO, which
+will use it to configure the OVN pods. The CNO has to watch for any change in this configuration.
+
+![Interaction between CNO and NOO](./ovs-flow-export-configuration-CNO-NOO.png)
+
+The `ConfigMap` content adds up with any configuration that is already present in the Network CR. For example, if a user has configured an IP for netflow exports in the Network CR AND has configured a Netobs CR for the NOO, OvS will export to both destinations.
+
+Here is the new `ConfigMap` definition:
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: ovs-flows-config
+  namespace: # CNO's namespace
+spec:
+  ipfix:
+    sharedTarget: <service ip+port>
+    nodePort: <int>
+    sampling: <int>
+    cacheActiveTimeout: <int (seconds)>
+    cacheMaxFlows: <int>
+```
+
+Assuming that each OVS instance will forward flows to the same node (to minimize network traffic
+between nodes), the `nodePort` property will tell the CNO which port of its node should send data
+to. The node IP is discoverable by OVN pods or the CNO.
+
+`nodePort` and `sharedTarget` are mutually exclusive: the former will be set by the NOO when the collector is deployed as a DaemonSet, the latter will be set when deployed as Deployment + Service. So, when `sharedTarget` is used, the CNO will simply need to copy it to the ovn-k pods environment variable for IPFIX export that already exists.
+
+Other [performance-tuning configuration](#tuning-flows-emission-parameters) properties should be also
+configurable and exposed, so the CNO or OVN-Kubernetes can properly
+[configure OpenVSwitch when it enables the flows](https://github.com/ovn-org/ovn-kubernetes/commit/ecc4047f5d0e732c267fccb2eabd64b5894b9f9a#diff-9c3b7372332fd16d60dba539a94dd0f819d262d1fbabd1e5d9c09e47da5af0ffR167). It's the `sampling`, `cacheActiveTimeout` and `cacheMaxFlows` defined in the `ConfigMap`. See also the [ovs-vsctl doc](http://www.openvswitch.org/support/dist-docs/ovs-vsctl.8.txt).
+
+
+#### Task 2: implement new discovery options in OVN-Kubernetes and OpenVswitch configuration
+
+Before [OVN-Kubernetes invokes OpenVSwitch to enable flows](https://github.com/ovn-org/ovn-kubernetes/commit/ecc4047f5d0e732c267fccb2eabd64b5894b9f9a#diff-9c3b7372332fd16d60dba539a94dd0f819d262d1fbabd1e5d9c09e47da5af0ffR119-R128),
+if the collector's node port environment variable is present (provided by the CNO as described in the previous section), it needs to retrieve its host IP and pass ip+port to OpenVSwitch, in the `targets=` argument.
+
+
+### Risks and Mitigations
+
+#### Conflicting configurations between the Network CR and the new shared ConfigMap
+
+At the moment there is no such direct conflict of settings, as there's no overlap. In Network CR, only the target IPs are configurable, and they would add to new IP configured by the NOO. However, two things are worth noting:
+
+1. The advanced settings offered via the NOO, like sampling and cache config, would impact all targets, including targets defined by the Network CR. This might not be expected by the user.
+
+There's no real mitigation for this, apart from documenting it as a limitation. We can however estimate that most users will not try to use concurrently flow exports (configured in the Network CR) and the NOO, as we can expect that those who manually configure flow exports already have their own monitoring infrastructure and dashboards in place for flows.
+
+2. It could slightly hinder future development precisely if new features are added in the Network CR related to IPFIX exports, that could possibly conflict with the ones defined in the `ConfigMap`. We would have then to find a way to prioritize or combine the two sources, or revisit this approach.
+
+## Design Details
+
+### Test Plan
+
+* Unit tests for all the components.
+* End-to-end tests:
+  * deploying a cluster with the CNO configured with ovn-k
+  * configure and deploy NOO with collector as a DaemonSet
+  * verify that it is collecting
+  * (etc.)
+
+### Graduation Criteria
+
+**Note:** *Section not required until targeted at a release.*
+
+#### Dev Preview -> Tech Preview
+#### Tech Preview -> GA
+#### Removing a deprecated feature
+
+### Upgrade / Downgrade Strategy
+
+There is no API change in the CNO, no new workload, so no particular risk in terms of the CNO upgrade / downgrade.
+
+If the CNO is downgraded, users of the NOO would loose the ability to have an automatic
+configuration of the flow exports from the NOO, as well as the ability to deploy the collector
+as a DaemonSet. As a workaround, they will still be able to rely on manual CNO configuration
+and deploy the flow collector as a Deployment + Service.
+
+Note that this isn't such an issue as the NOO is new and will start as a dev preview.
+
+### Version Skew Strategy
+
+We don't envision issues with respect to version skew.
+
+## Implementation History
+
+Major milestones in the life cycle of a proposal should be tracked in `Implementation
+History`.
+
+## Drawbacks
+
+To be discussed.
+
+## Alternatives
+
+* Rely on manual configuration of the NOO and the CNO separately. Stick with a single deployment model available for the collector, as Deployment + Service.
+
+* Having ovn-k reading and watching for the new OvS `ConfigMap`, instead of the CNO. It would remove the soft dependency introduced between CNO and NOO, however we think it's more the job of an operator to watch for config changes.