Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SDN 2316: Use GatewayConfig in OVN-K to set gateway modes #1209

Merged
merged 2 commits into from
Dec 14, 2021
+72 −89
Merged

SDN 2316: Use GatewayConfig in OVN-K to set gateway modes #1209

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0f38cc7
Remove exclusiveness between egressIPs and ICNI
tssurya Dec 6, 2021
5ee9f36
Respect GatewayConfig values
tssurya Dec 6, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,10 +195,14 @@ data:
```

### Configuring OVNKubernetes
OVNKubernetes supports the following configuration options, all of which are optional and once set at cluster creation, they can't be changed:
OVNKubernetes supports the following configuration options, all of which are optional and once set at cluster creation, they can't be changed except for `gatewayConfig` which can be changed at runtime:
* `MTU`: The MTU to use for the geneve overlay. The default is the MTU of the node that the cluster-network-operator is first run on, minus 100 bytes for geneve overhead. If the nodes in your cluster don't all have the same MTU then you may need to set this explicitly.
* `genevePort`: The UDP port to use for the Geneve overlay. The default is 6081.
* `hybridOverlayConfig`: hybrid linux/windows cluster (see below).
* `ipsecConfig`: enables and configures IPsec for pods on the pod network within the cluster.
* `policyAuditConfig`: holds the configuration for network policy audit events.
* `gatewayConfig`: holds the configuration for node gateway options.
* `routingViaHost`: If set to true, pod egress traffic will touch host networking stack before being sent out.

These configuration flags are only in the Operator configuration object.

Expand All @@ -210,6 +214,8 @@ spec:
ovnKubernetesConfig:
mtu: 1400
genevePort: 6081
gatewayConfig:
routingViaHost: false
```

Additionally, you can configure per-node verbosity for ovn-kubernetes. This is useful
Expand Down
2 changes: 0 additions & 2 deletions bindata/network/ovn-kubernetes/004-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,7 @@ data:
{{- end }}

[ovnkubernetesfeature]
{{- if .OVN_ENABLE_EGRESS_IP}}
enable-egress-ip=true
{{- end}}
enable-egress-firewall=true

[gateway]
Expand Down
3 changes: 0 additions & 3 deletions bindata/network/ovn-kubernetes/ovnkube-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -709,9 +709,6 @@ spec:
--nbctl-daemon-mode \
--nb-cert-common-name "{{.OVN_CERT_CN}}" \
--enable-multicast \
{{- if .OVN_DISABLE_SNAT_MULTIPLE_GWS }}
--disable-snat-multiple-gws \
{{- end }}
--acl-logging-rate-limit "{{.OVNPolicyAuditRateLimit}}"
lifecycle:
preStop:
Expand Down
3 changes: 0 additions & 3 deletions bindata/network/ovn-kubernetes/ovnkube-node.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -314,9 +314,6 @@ spec:
{{- end }}
--metrics-bind-address "127.0.0.1:29103" \
--metrics-enable-pprof \
{{- if .OVN_DISABLE_SNAT_MULTIPLE_GWS }}
--disable-snat-multiple-gws \
{{- end }}
${export_network_flows_flags} \
${gw_interface_flag}
env:
Expand Down
4 changes: 0 additions & 4 deletions hack/KIND_CNO.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,10 +79,6 @@ $ IP_FAMILY=ipv6 ovn-kind-cno.sh
This will build an OVN K8S docker image to use with the deployment from a local git workspace. By default the script will use
a path relative to your GOPATH. To override this, specify `CNO_PATH` or `OVN_K8S_PATH` when executing the above.

If there is a desire to deploy with `local` gateway mode (default is `shared`), you can modify the `ovn-kind-cno.sh`
script to create a configMap using the example found [here](./gateway-mode.yaml). Do this as a step right after the
cluster-config-v1 configMap is created.

### Post deployment

Post deployment you can simply use kubectl to interact with your cluster. Additionally you may wish to exec into the
Expand Down
10 changes: 0 additions & 10 deletions hack/gateway-mode.yaml
View file
Open in desktop

This file was deleted.

6 changes: 2 additions & 4 deletions pkg/bootstrap/types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,8 @@ type KuryrBootstrapResult struct {
}

type OVNConfigBoostrapResult struct {
GatewayMode string
NodeMode string
EnableEgressIP bool
DisableSNATMultipleGWs bool
GatewayMode string
NodeMode string
}

type OVNBootstrapResult struct {
Expand Down
78 changes: 42 additions & 36 deletions pkg/network/ovn_kubernetes.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,10 +82,7 @@ func renderOVNKubernetes(conf *operv1.NetworkSpec, bootstrapResult *bootstrap.Bo
data.Data["GenevePort"] = c.GenevePort
data.Data["CNIConfDir"] = pluginCNIConfDir(conf)
data.Data["CNIBinDir"] = CNIBinDir
data.Data["OVN_GATEWAY_MODE"] = bootstrapResult.OVN.OVNKubernetesConfig.GatewayMode
data.Data["OVN_NODE_MODE"] = OVN_NODE_MODE_FULL
data.Data["OVN_ENABLE_EGRESS_IP"] = bootstrapResult.OVN.OVNKubernetesConfig.EnableEgressIP
data.Data["OVN_DISABLE_SNAT_MULTIPLE_GWS"] = bootstrapResult.OVN.OVNKubernetesConfig.DisableSNATMultipleGWs
data.Data["OVN_NB_PORT"] = OVN_NB_PORT
data.Data["OVN_SB_PORT"] = OVN_SB_PORT
data.Data["OVN_NB_RAFT_PORT"] = OVN_NB_RAFT_PORT
Expand Down Expand Up @@ -161,6 +158,12 @@ func renderOVNKubernetes(conf *operv1.NetworkSpec, bootstrapResult *bootstrap.Bo
data.Data["EnableIPsec"] = false
}

if c.GatewayConfig != nil && c.GatewayConfig.RoutingViaHost {
data.Data["OVN_GATEWAY_MODE"] = OVN_LOCAL_GW_MODE
} else {
data.Data["OVN_GATEWAY_MODE"] = OVN_SHARED_GW_MODE
}

exportNetworkFlows := conf.ExportNetworkFlows
if exportNetworkFlows != nil {
if exportNetworkFlows.NetFlow != nil {
Expand Down Expand Up @@ -279,43 +282,16 @@ func renderOVNKubernetes(conf *operv1.NetworkSpec, bootstrapResult *bootstrap.Bo
return objs, nil
}

// bootstrapOVNConfig returns the value of mode found in the openshift-ovn-kubernetes/gateway-mode-config configMap
// bootstrapOVNConfig returns the value of mode found in the openshift-ovn-kubernetes/dpu-mode-config configMap
// if it exists, otherwise returns default configuration for OCP clusters using OVN-Kubernetes
func bootstrapOVNConfig(kubeClient client.Client) (*bootstrap.OVNConfigBoostrapResult, error) {
func bootstrapOVNConfig(conf *operv1.Network, kubeClient client.Client) (*bootstrap.OVNConfigBoostrapResult, error) {
ovnConfigResult := &bootstrap.OVNConfigBoostrapResult{
GatewayMode: OVN_SHARED_GW_MODE,
NodeMode: OVN_NODE_MODE_FULL,
EnableEgressIP: true,
DisableSNATMultipleGWs: false,
NodeMode: OVN_NODE_MODE_FULL,
}

bootstrapOVNGatewayConfig(conf, kubeClient)
cm := &corev1.ConfigMap{}
nsn := types.NamespacedName{Namespace: "openshift-network-operator", Name: "gateway-mode-config"}
err := kubeClient.Get(context.TODO(), nsn, cm)

if err != nil {
if apierrors.IsNotFound(err) {
klog.Infof("Did not find gateway-mode-config")
} else {
return nil, fmt.Errorf("Could not determine gateway mode: %w", err)
}
} else {
modeOverride := cm.Data["mode"]
_, disableSNATMultipleGWsOverride := cm.Data["disable-snat-multiple-gws"]
if modeOverride != OVN_SHARED_GW_MODE && modeOverride != OVN_LOCAL_GW_MODE {
klog.Warningf("gateway-mode-config does not match %q or %q, is: %q. Using default OVN configuration: %+v", OVN_LOCAL_GW_MODE, OVN_SHARED_GW_MODE, modeOverride, ovnConfigResult)
return ovnConfigResult, nil
}
ovnConfigResult.GatewayMode = modeOverride
if disableSNATMultipleGWsOverride {
ovnConfigResult.EnableEgressIP = false
ovnConfigResult.DisableSNATMultipleGWs = true
}
klog.Infof("Overriding OVN configuration to %+v", ovnConfigResult)
}

dmc := types.NamespacedName{Namespace: "openshift-network-operator", Name: "dpu-mode-config"}
err = kubeClient.Get(context.TODO(), dmc, cm)
err := kubeClient.Get(context.TODO(), dmc, cm)

if err != nil {
if apierrors.IsNotFound(err) {
Expand Down Expand Up @@ -482,6 +458,36 @@ type replicaCountDecoder struct {
} `json:"controlPlane"`
}

// bootstrapOVNGatewayConfig sets the Network.operator.openshift.io.Spec.DefaultNetwork.OVNKubernetesConfig.GatewayConfig value
// based on the values from the "gateway-mode-config" map if any
func bootstrapOVNGatewayConfig(conf *operv1.Network, kubeClient client.Client) {
// handle upgrade logic for gateway mode in OVN-K plugin (migration from hidden config map to using proper API)
// TODO: Remove this logic in future releases when we are sure everyone has migrated away from the config-map
cm := &corev1.ConfigMap{}
nsn := types.NamespacedName{Namespace: "openshift-network-operator", Name: "gateway-mode-config"}
err := kubeClient.Get(context.TODO(), nsn, cm)
modeOverride := OVN_SHARED_GW_MODE
routeViaHost := false

if err != nil {
klog.Infof("Did not find gateway-mode-config. Using default gateway mode: %s", OVN_SHARED_GW_MODE)
} else {
modeOverride = cm.Data["mode"]
if modeOverride != OVN_SHARED_GW_MODE && modeOverride != OVN_LOCAL_GW_MODE {
klog.Warningf("gateway-mode-config does not match %q or %q, is: %q. Using default gateway mode: %s",
OVN_LOCAL_GW_MODE, OVN_SHARED_GW_MODE, modeOverride, OVN_SHARED_GW_MODE)
modeOverride = OVN_SHARED_GW_MODE
}
}
if modeOverride == OVN_LOCAL_GW_MODE {
routeViaHost = true
}
conf.Spec.DefaultNetwork.OVNKubernetesConfig.GatewayConfig = &operv1.GatewayConfig{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't actually update the config map right? Do we need to update the config map from here so that when we remove the ability to use the hidden configmap we dont break users? Or should we print some warning or event to tell the user they need to update the gatewayConfig in their configmap?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do update the configmap when this function returns. Let me explain:

  1. cluster-network-operator/pkg/network/bootstrap.go

    Line 18 in 38b07c3

    case operv1.NetworkTypeOVNKubernetes:

  2. cluster-network-operator/pkg/network/ovn_kubernetes.go

    Line 485 in 38b07c3

    func bootstrapOVN(conf *operv1.Network, kubeClient client.Client) (*bootstrap.BootstrapResult, error) {

  3. cluster-network-operator/pkg/network/ovn_kubernetes.go

    Line 284 in 38b07c3

    func bootstrapOVNConfig(kubeClient client.Client) (*bootstrap.OVNConfigBoostrapResult, error) {
  4. Within bootstrapOVNConfig we do our thing.
  5. Then we return all the way back to the bootstrap call

    cluster-network-operator/pkg/controller/operconfig/operconfig_controller.go

    Line 192 in 38b07c3

    bootstrapResult, err := network.Bootstrap(newOperConfig, r.client)
  6. Aaaand this is where we update the CRD:

    cluster-network-operator/pkg/controller/operconfig/operconfig_controller.go

    Lines 200 to 201 in 38b07c3

    if !reflect.DeepEqual(operConfig, newOperConfig) {
    if err := r.UpdateOperConfig(newOperConfig); err != nil {

Hence any changes done to the CRD during bootstrap gets written into the CRD. After all that the rendering starts and it regards the updated CRD the source of truth.

I tested it using the steps mentioned in https://docs.google.com/document/d/1SYaNQNW4sWVTE9XgHG6gKsS4ZActK8_Z7wILXU5YuoU/edit#heading=h.d5s838mpuww0 in a cluster-bot cluster and it works well. The CRD does get updated to something like:

Gateway Config:{
    Routing Via Host: true
}

RoutingViaHost: routeViaHost,
}
klog.Infof("Gateway mode is %s", modeOverride)
}

func bootstrapOVN(conf *operv1.Network, kubeClient client.Client) (*bootstrap.BootstrapResult, error) {
clusterConfig := &corev1.ConfigMap{}
clusterConfigLookup := types.NamespacedName{Name: CLUSTER_CONFIG_NAME, Namespace: CLUSTER_CONFIG_NAMESPACE}
Expand All @@ -508,7 +514,7 @@ func bootstrapOVN(conf *operv1.Network, kubeClient client.Client) (*bootstrap.Bo
return nil, fmt.Errorf("Unable to bootstrap OVN, unable to unmarshal install-config: %s", err)
}

ovnConfigResult, err := bootstrapOVNConfig(kubeClient)
ovnConfigResult, err := bootstrapOVNConfig(conf, kubeClient)
if err != nil {
return nil, fmt.Errorf("Unable to bootstrap OVN config, err: %v", err)
}
Expand Down
47 changes: 21 additions & 26 deletions pkg/network/ovn_kubernetes_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,7 @@ func TestRenderOVNKubernetes(t *testing.T) {
OVN: bootstrap.OVNBootstrapResult{
MasterIPs: []string{"1.2.3.4", "5.6.7.8", "9.10.11.12"},
OVNKubernetesConfig: &bootstrap.OVNConfigBoostrapResult{
GatewayMode: "shared",
NodeMode: "full",
EnableEgressIP: true,
DisableSNATMultipleGWs: false,
NodeMode: "full",
},
},
}
Expand Down Expand Up @@ -131,10 +128,7 @@ func TestRenderOVNKubernetesIPv6(t *testing.T) {
OVN: bootstrap.OVNBootstrapResult{
MasterIPs: []string{"1.2.3.4", "5.6.7.8", "9.10.11.12"},
OVNKubernetesConfig: &bootstrap.OVNConfigBoostrapResult{
GatewayMode: "shared",
NodeMode: "full",
EnableEgressIP: true,
DisableSNATMultipleGWs: false,
NodeMode: "full",
},
},
}
Expand All @@ -150,10 +144,7 @@ func TestRenderOVNKubernetesIPv6(t *testing.T) {
OVN: bootstrap.OVNBootstrapResult{
MasterIPs: []string{"fd01::1", "fd01::2", "fd01::3"},
OVNKubernetesConfig: &bootstrap.OVNConfigBoostrapResult{
GatewayMode: "shared",
NodeMode: "full",
EnableEgressIP: true,
DisableSNATMultipleGWs: false,
NodeMode: "full",
},
},
}
Expand All @@ -171,6 +162,7 @@ func TestRenderedOVNKubernetesConfig(t *testing.T) {
desc string
expected string
hybridOverlayConfig *operv1.HybridOverlayConfig
gatewayConfig *operv1.GatewayConfig
masterIPs []string
}
testcases := []testcase{
Expand Down Expand Up @@ -221,7 +213,7 @@ enable-egress-ip=true
enable-egress-firewall=true

[gateway]
mode=shared
mode=local
nodeport=true

[hybridoverlay]
Expand All @@ -232,6 +224,9 @@ cluster-subnets="10.132.0.0/14"`,
{CIDR: "10.132.0.0/14", HostPrefix: 23},
},
},
gatewayConfig: &operv1.GatewayConfig{
RoutingViaHost: true,
},
masterIPs: []string{"1.2.3.4", "2.3.4.5"},
},
{
Expand All @@ -256,7 +251,7 @@ enable-egress-ip=true
enable-egress-firewall=true

[gateway]
mode=shared
mode=local
nodeport=true

[hybridoverlay]
Expand All @@ -270,6 +265,9 @@ hybrid-overlay-vxlan-port="9000"`,
},
HybridOverlayVXLANPort: ptrToUint32(9000),
},
gatewayConfig: &operv1.GatewayConfig{
RoutingViaHost: true,
},
masterIPs: []string{"1.2.3.4", "2.3.4.5"},
},
{
Expand Down Expand Up @@ -332,6 +330,9 @@ election-lease-duration=137
election-renew-deadline=107
election-retry-period=26`,
masterIPs: []string{"1.2.3.4"},
gatewayConfig: &operv1.GatewayConfig{
RoutingViaHost: false,
},
},
}
g := NewGomegaWithT(t)
Expand All @@ -345,6 +346,9 @@ election-retry-period=26`,
if tc.hybridOverlayConfig != nil {
OVNKubeConfig.Spec.DefaultNetwork.OVNKubernetesConfig.HybridOverlayConfig = tc.hybridOverlayConfig
}
if tc.hybridOverlayConfig != nil {
OVNKubeConfig.Spec.DefaultNetwork.OVNKubernetesConfig.GatewayConfig = tc.gatewayConfig
}

//set a few inputs so that the tests are not machine dependant
OVNKubeConfig.Spec.DefaultNetwork.OVNKubernetesConfig.MTU = ptrToUint32(1500)
Expand All @@ -360,10 +364,7 @@ election-retry-period=26`,
OVN: bootstrap.OVNBootstrapResult{
MasterIPs: tc.masterIPs,
OVNKubernetesConfig: &bootstrap.OVNConfigBoostrapResult{
GatewayMode: "shared",
NodeMode: "full",
EnableEgressIP: true,
DisableSNATMultipleGWs: false,
NodeMode: "full",
},
},
}
Expand Down Expand Up @@ -1234,10 +1235,7 @@ metadata:
ExistingMasterDaemonset: master,
ExistingNodeDaemonset: node,
OVNKubernetesConfig: &bootstrap.OVNConfigBoostrapResult{
GatewayMode: "shared",
NodeMode: "full",
EnableEgressIP: true,
DisableSNATMultipleGWs: false,
NodeMode: "full",
},
PrePullerDaemonset: prepuller,
},
Expand Down Expand Up @@ -1537,10 +1535,7 @@ func TestRenderOVNKubernetesDualStackPrecedenceOverUpgrade(t *testing.T) {
},
},
OVNKubernetesConfig: &bootstrap.OVNConfigBoostrapResult{
GatewayMode: "shared",
NodeMode: "full",
EnableEgressIP: true,
DisableSNATMultipleGWs: false,
NodeMode: "full",
},
},
}
Expand Down