Introduce DPU Network Operator #890

pliurh · 2021-09-07T13:29:52Z

Signed-off-by: Peng Liu

enhancements/network/dpu-ovnkube-operator.md

danwinship · 2021-09-13T20:19:49Z

enhancements/network/dpu-ovnkube-operator.md

+
+The DPU OVNKube Operator will be responsible for enable the switchdev mode on
+BF2 cards. We assume all the BF2 cards will be put into a custom
+MachineConfigPool,


comma should be a period, or else there's something missing at the end?

Who is responsible for the custom MachineConfigPool? When are they supposed to create it?

Who is responsible for the custom MachineConfigPool? When are they supposed to create it?

That is a good question. In my assumption, the cluster-admin shall label the nodes and create the MCP. WDYT?

enhancements/network/dpu-ovnkube-operator.md

danwinship · 2021-09-13T20:27:37Z

enhancements/network/dpu-ovnkube-operator.md

+
+We will create a namespaced CRD OVNKubeConfig.smartnic.openshift.io for the
+DPU OVNKube Operator. One custom resource of this CRD shall be created for
+the tenant cluster. The operator will take this custom resource as input to


it's created "for the tenant cluster", but it's created in the infra cluster, which again probably needs to be clarified.

How is the admin expected to create this resource? We should probably provide at least a script or something to grab the necessary information from the tenant cluster and create the resource in the infra cluster...

I don't quite get the question. This operator only needs the kubeconfig of the tenant cluster, then it can access the apiserver to grab all the info. Do you mean we need a script to generate the kubeconfig file for the tenant cluster?

In my assumption, the whole process shall be like this:

After infra cluster installation, the admin labels the DPUs which he wants to enable hardware offload.

The admin creates a custom MCP that contains all the selected DPUs.

The admin configures the CNO to deprovision the ovn-kube components from the DPUs.

The admin installs the DPU operator and creates an OVNKubeConfig CR with the poolName but leaves kubeConfigFile blank. The operator then does the host network configuration.

Install the tenant cluster. Scale up the x86 workers with DPU.

The CNO of the tenant cluster puts those x86 hosts into smart-nic host mode.

The admin configures VFs on x86 node with the Sriov Network Operator.

The admin stores the kubeconfig file in the infra cluster then adds kubeConfigFile to the existing OVNKubeConfig CR. The operator installs the ovn-kube components on the selected DPUs.

enhancements/network/dpu-ovnkube-operator.md

zshi-redhat · 2021-09-15T01:58:29Z

enhancements/network/dpu-ovnkube-operator.md

+  - enable the switchdev mode on DPUs.
+  - add PF representor to the br-ex bridge, so that the x86 host can connect to
+  the cluster network of the tenant cluster.
+- Define the ovn-kube upgrade mechanism in such a deployment.


I don't see the ovn-kube upgrade mechanism is discussed in this doc (we talked about the ovnkube operator upgrade in the lifecycle management section).
How can we ensure the version of ovnkube-node in infra cluster is compatible with the version of ovn-kube components in tenant cluster during cluster upgrade or during ovnkube operator installation?

For example, should this OVNKube operator fetch the image information from tenant cluster as well and use the image to spin up ovnkube-node in infra cluster?

Can we assume the two clusters will always be in the same version, and upgrade together? Then we can let the operator use the images which are currently used in the infra cluster.

I think we need to define the version compatibility between infra and tenant cluster, like what is supported and not supported. For example, we may say that 4.x tenant is compatible with 4.x+1 infra or 4.x+n infra.

This would allow new feature be introduced without updating tenant cluster. for example, the update of rhcos image in infra cluster (openvswitch, driver etc) may bring in new features, Operator designed to run on infra cluster would also benefit from it.

Thinking loudly, if we were to move ovn-central to infra cluster, it would further decouple the infra and tenant in terms of cluster or ovnkube versions. which allow more features be added without updating tenant cluster.

zshi-redhat · 2021-09-15T02:05:26Z

enhancements/network/dpu-ovnkube-operator.md

+## Design Details
+
+The DPU OVNKube Operator will fetch the necessary information from the
+tenant cluster, then create the following objects in the infra cluster:


Shall we add that OVNKube operator will watch those runtime ovn-k8s changes in tenant cluster and reflect that accordingly in the infra clusters? Like whenever the configMap or Secret are changed in tenant cluster.

Yeah, it's a good point. AFAIK, those ovnkube objects can not be changed after cluster installation because CNO doesn't allow them to. But the certificates do expire, we need to watch the change in the tenant cluster.

One example I had in mind was the ipsec enablement, it can be done after initial cluster installation and ovnkube Operator will need to render the ipsec daemonset in infra cluster.

zshi-redhat · 2021-09-15T02:07:07Z

enhancements/network/dpu-ovnkube-operator.md

+
+## Design Details
+
+The DPU OVNKube Operator will fetch the necessary information from the


Does this Operator need to handle kube API compatibility between infra and tenant clusters?

Is that necessary? I assume both clusters shall in the same version.

If we allow tenant and infra to be in different cluster versions (which I think we should if not in the initial delivery), then yes.

enhancements/network/dpu-ovnkube-operator.md

Billy99 · 2021-09-15T19:12:36Z

enhancements/network/dpu-ovnkube-operator.md

+## Motivation
+
+Normally, the Cluster Network Operator watches the Network customer resource,
+then provision the ovn-kube components to the Openshift cluster.


Normally CNO manages the ovn-kube components. If this new operator is taking over that role for the infra cluster, do we need to document somewhere that CNO needs to be changed to not manage OVN-Kube on infra clusters and what's the CNO trigger for that?

This operator is not taking take over that role for the infra cluster, but managing the ovnkube that servers the tenant cluster. The CNO will still manage the ovnkube of the infra cluster on non-BF2 nodes.

@zshi-redhat Shall we mention these CNO changes in #732

I think we need a similar patch in CNO like this openshift/cluster-network-operator#1138, to isolate the CNO managed ovnkube from being scheduled on DPU in infra cluster. For example, to introduce the label network.operator.openshift.io/dpu which will be used by CNO in infra cluster.

@pliurh yes, THE #732 needs to be updated.

enhancements/network/dpu-ovnkube-operator.md

danwinship · 2021-11-03T18:51:06Z

can you move this into the enhancements/network/dpu/ directory (which should exist now in master)?

you may need to add some more dummy sections to make the latest markdownlint job happy too...

Signed-off-by: Peng Liu <Peng Liu>

pliurh · 2021-11-04T03:35:40Z

can you move this into the enhancements/network/dpu/ directory (which should exist now in master)?

you may need to add some more dummy sections to make the latest markdownlint job happy too...

Done

danwinship · 2021-11-04T15:41:24Z

/lgtm

danwinship · 2021-11-08T20:29:51Z

/approve

openshift-ci · 2021-11-08T20:30:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danwinship

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~enhancements/network/OWNERS~~ [danwinship]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci bot requested review from danwinship and dougbtv September 7, 2021 13:30

pliurh force-pushed the dpu-ovnkube-operator branch from bb7d3fe to e021d60 Compare September 7, 2021 13:55

danwinship reviewed Sep 13, 2021

View reviewed changes

pliurh force-pushed the dpu-ovnkube-operator branch 2 times, most recently from f918873 to 0eaa1c7 Compare September 14, 2021 07:09

zshi-redhat reviewed Sep 15, 2021

View reviewed changes

pliurh force-pushed the dpu-ovnkube-operator branch from 0eaa1c7 to 16af993 Compare September 15, 2021 06:59

Billy99 reviewed Sep 15, 2021

View reviewed changes

danwinship mentioned this pull request Sep 21, 2021

Installing OCP on ARM-Based Smart NICs #738

Merged

bn222 reviewed Oct 14, 2021

View reviewed changes

enhancements/network/dpu-ovnkube-operator.md Outdated Show resolved Hide resolved

pliurh force-pushed the dpu-ovnkube-operator branch from d052542 to 52d1b2e Compare November 4, 2021 03:11

Introduce DPU OVNKube Operator

6caff80

Signed-off-by: Peng Liu <Peng Liu>

pliurh force-pushed the dpu-ovnkube-operator branch from 52d1b2e to 6caff80 Compare November 4, 2021 03:28

pliurh changed the title ~~Introduce DPU OVNKube Operator~~ Introduce DPU Network Operator Nov 4, 2021

openshift-ci bot assigned danwinship Nov 4, 2021

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 4, 2021

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 8, 2021

openshift-merge-robot merged commit cf1d902 into openshift:master Nov 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce DPU Network Operator #890

Introduce DPU Network Operator #890

pliurh commented Sep 7, 2021

danwinship Sep 13, 2021

pliurh Sep 14, 2021

danwinship Sep 13, 2021

pliurh Sep 14, 2021 •

edited

Loading

zshi-redhat Sep 15, 2021

zshi-redhat Sep 15, 2021

pliurh Sep 15, 2021

zshi-redhat Sep 16, 2021

zshi-redhat Sep 15, 2021

pliurh Sep 15, 2021

zshi-redhat Sep 16, 2021 •

edited

Loading

zshi-redhat Sep 15, 2021

pliurh Sep 15, 2021

zshi-redhat Sep 16, 2021 •

edited

Loading

Billy99 Sep 15, 2021

pliurh Sep 30, 2021

zshi-redhat Sep 30, 2021

zshi-redhat Sep 30, 2021

danwinship commented Nov 3, 2021

pliurh commented Nov 4, 2021

danwinship commented Nov 4, 2021

danwinship commented Nov 8, 2021

openshift-ci bot commented Nov 8, 2021


		## Design Details

		The DPU OVNKube Operator will fetch the necessary information from the

Introduce DPU Network Operator #890

Introduce DPU Network Operator #890

Conversation

pliurh commented Sep 7, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pliurh Sep 14, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zshi-redhat Sep 16, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zshi-redhat Sep 16, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danwinship commented Nov 3, 2021

pliurh commented Nov 4, 2021

danwinship commented Nov 4, 2021

danwinship commented Nov 8, 2021

openshift-ci bot commented Nov 8, 2021

pliurh Sep 14, 2021 •

edited

Loading

zshi-redhat Sep 16, 2021 •

edited

Loading

zshi-redhat Sep 16, 2021 •

edited

Loading