Skip to content

Conversation

@bpickard22
Copy link
Contributor

We currently reserve a range of addresses to configure host to service traffic internally in ovn. We need to allow users to specify this range to avoid conflicting with addresses they use in their local infra

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 17, 2023

Hello @bpickard22! Some important instructions when contributing to openshift/api:
API design plays an important part in the user experience of OpenShift and as such API PRs are subject to a high level of scrutiny to ensure they follow our best practices. If you haven't already done so, please review the OpenShift API Conventions and ensure that your proposed changes are compliant. Following these conventions will help expedite the api review process for your PR.

@openshift-ci openshift-ci bot requested review from knobunc and soltysh February 17, 2023 14:39
@abhat
Copy link
Contributor

abhat commented Feb 17, 2023

LGTM

@abhat
Copy link
Contributor

abhat commented Feb 17, 2023

need to do a codegen since the verify job is failing?

Copy link
Contributor

@tssurya tssurya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left a few comments, but I know we are waiting for the bits from the generated code..

//The host is configured with these addresses, as well as the shared gateway
//bridge interface. The values can be changed after installation.
//The defaults are 169.254.169.1 - 169.254.169.4 and fd69::1 - fd69::4
//+optional
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: These are kubebuilder formats, please follow convention and put the comment above the field.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Example start the comment with gatewaySubnet contains.. blah blah...

v6InternalSubnet:
description: v6InternalSubnet is a v6 subnet used internally by ovn-kubernetes in case the default one is being already used by something else. It must not overlap with any other subnet being used by OpenShift or by the node network. The size of the subnet must be larger than the number of nodes. The value cannot be changed after installation. Default is fd98::/48
type: string
GatewaySubnet:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did you manually edit this or did this get auto-generated? gatewaySubnet is what I'd expect...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did that manually

// egressIPConfig holds the configuration for EgressIP options.
// +optional
EgressIPConfig EgressIPConfig `json:"egressIPConfig,omitempty"`
GatewaySubnet string `json:"GatewaySubnet,omitempty"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah I see why now..

Suggested change
GatewaySubnet string `json:"GatewaySubnet,omitempty"`
GatewaySubnet string `json:"gatewaySubnet,omitempty"`

please pay attention to semantics, notice how other fields are being done..

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will correct

// egressIPConfig holds the configuration for EgressIP options.
// +optional
EgressIPConfig EgressIPConfig `json:"egressIPConfig,omitempty"`
GatewaySubnet string `json:"GatewaySubnet,omitempty"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestions:
Should we put these new options inside the GatewayConfig?
And shouldn't we have 2 subnet options one for v4 and another for v6 ? How do you plan to have both of them in this single field?

169.254.169.0/16 isn't exactly a Gateway Subnet, if we move this into GatewayConfig then we could call them V4InternalMasqueradeSubnet and V6InternalMasqueradeSubnet..

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes I think this makes more sense, will use this approach

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @trozet if he is ok with this.

Copy link
Contributor

@tssurya tssurya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there.
BTW this can't merge without the OVNK PR right? Probably best to put a hold until then?

// ovn-kubernetes to enable host to service traffic. The host is configured with these
// addresses, as well as the shared gateway bridge interface.The values can be changed after
// installation.
// The defaults are 169.254.169.1 - 169.254.169.4
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We also use the 169.254.169.5 btw for the hairpin service traffic. Since its a subnet, its enough to say
Default is 169.254.169.0/16 or whatever is the default "subnet".
Doesn't make sense to quote the IPs within the subnet as defaults right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there. BTW this can't merge without the OVNK PR right? Probably best to put a hold until then?

right, just wanted this up for reivew

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tssurya should we just call it a range then. If the user needs to use those addresses for their infra, are we going to shift the masq addresses to a different subnet or just silde them down in the same subnet?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should be a subnet. The defaults right now are:
V4MasqueradeSubnet = "169.254.169.0/29"
V6MasqueradeSubnet = "fd69::/125"

We can specify that the minimum cidr needs to be at least /29 for v4 and at least /125 for v6

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@trozet is there a reason that we need to specify that min cidr?

// ovn-kubernetes to enable host to service traffic. The host is configured with these
// addresses, as well as the shared gateway bridge interface.The values can be changed after
// installation.
// The defaults are fd69::1 - fd69::4
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

// egressIPConfig holds the configuration for EgressIP options.
// +optional
EgressIPConfig EgressIPConfig `json:"egressIPConfig,omitempty"`
GatewaySubnet string `json:"GatewaySubnet,omitempty"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @trozet if he is ok with this.

@trozet
Copy link
Contributor

trozet commented Feb 27, 2023

I thought we also need the ability to specify the join subnet?

@bpickard22 bpickard22 force-pushed the gatway-subnet-api branch from 2b49375 to 3e7785f Compare May 4, 2023 21:00
@openshift-ci openshift-ci bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label May 4, 2023
@trozet
Copy link
Contributor

trozet commented May 5, 2023

I thought we also need the ability to specify the join subnet?

@bpickard22 pointed out it already exists via: #1207

@trozet
Copy link
Contributor

trozet commented May 5, 2023

/lgtm

@trozet
Copy link
Contributor

trozet commented May 5, 2023

/hold

until OVNK bits land

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 5, 2023
@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label May 5, 2023
@openshift-ci openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Aug 4, 2023
@trozet
Copy link
Contributor

trozet commented Aug 7, 2023

/hold cancel

@trozet
Copy link
Contributor

trozet commented Aug 7, 2023

/lgtm

@openshift-ci openshift-ci bot added lgtm Indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Aug 7, 2023
@trozet
Copy link
Contributor

trozet commented Aug 7, 2023

/approve

// +optional
IPForwarding IPForwardingMode `json:"ipForwarding,omitempty"`
// V4InternalMasqueradeSubnet contains the v4 masquerade addresses used internally by
// ovn-kubernetes to enable host to service traffic. The host is configured with these
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which host?

Copy link
Contributor Author

@bpickard22 bpickard22 Aug 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These addresses would be configured on all hosts on the network, as they all use the same masq subnet to enable the host to service traffic flow described here https://github.com/ovn-org/ovn-kubernetes/blob/master/docs/design/host_to_services_OpenFlow.md

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

specifically, addresses from this subnet are configured on every OCP node, including as secondary IP addresses on host linux interfaces and as well as addresses assigned to internal OVN interfaces

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should update the description, perhaps Each host in the cluster is configured... covers this?

// The supported values are "Restricted" and "Global".
// +optional
IPForwarding IPForwardingMode `json:"ipForwarding,omitempty"`
// V4InternalMasqueradeSubnet contains the v4 masquerade addresses used internally by
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Under what conditions would a cluster-admin want to change this and how would they know if they were choosing a "good" value? How would they know if their choice was bad?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A cluster admin would want to change this if they want to addresses that we currently use here in their local infrastructure. I think "good" may be subjective, but any address range that neither we nor the cloud provider reserve should be good

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can just say: The subnet chosen should not overlap with other networks specified for OVN-Kubernetes as well as other networks used on the host. Additionally the subnet must be large enough to accommodate 6 IPs (maximum prefix length /29).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the description with this addition

// ovn-kubernetes to enable host to service traffic. The host is configured with these
// addresses, as well as the shared gateway bridge interface.The values can be changed after
// installation.
// The default subnet is 169.254.169.0/29
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's a regex that can validate this. I bet @Miciah knows for this and for the ipv6 one.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we dont currently use regex to validate the other configurable subnet in ovn-k (the internal join subnet), and any invalid value passed in would be caught by the cno or in ovnk, do you think we need that here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added regex taken from

// CIDR is an IP address range in CIDR notation (for example, "10.0.0.0/8"
// or "fd00::/8").
// +kubebuilder:validation:Pattern=`(^(([0-9]|[0-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[0-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])/([0-9]|[12][0-9]|3[0-2])$)|(^s*((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-4]d|1dd|[1-9]?d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-4]d|1dd|[1-9]?d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-4]d|1dd|[1-9]?d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-4]d|1dd|[1-9]?d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-4]d|1dd|[1-9]?d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-4]d|1dd|[1-9]?d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-4]d|1dd|[1-9]?d)){3}))|:)))(%.+)?s*(\/(12[0-8]|1[0-1][0-9]|[1-9][0-9]|[0-9]))$)`
// + ---
// + The regex for the IPv4 CIDR range was taken from other CIDR fields in the OpenShift API
// + and the one for the IPv6 CIDR range was taken from
// + https://blog.markhatton.co.uk/2011/03/15/regular-expressions-for-ip-addresses-cidr-ranges-and-hostnames/
// + The resulting regex is an OR of both regexes.
type CIDR string

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

split into v4 and v6 respectively

Copy link
Contributor

@JoelSpeed JoelSpeed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of nits to fix, but otherwise LGTM. I would appreciate if the networking team could ack the integration tests for the validations here, especially for the v6, we need to be sure we are validating the requirements of a v6 address correctly as we won't be able to change this once released.

gatewayConfig:
ipv6:
internalMasqueradeSubnet: "abcd:ef01:2345:6789:abcd:ef01:2345::/125"
expectedError: "Invalid value: \"string\": a valid IPv6 address must contain 8 segments unless elided (::), in which case it must contain at most 6 non-empty segments"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, missing new line at the end of this file

// +kubebuilder:validation:XValidation:rule="[self.findAll('[0-9]+')[0]].all(x, x != '0' && int(x) <= 255 && !x.startsWith('0'))",message="first IP address octet must not contain leading zeros, must be greater than 0 and less or equal to 255"
// +kubebuilder:validation:XValidation:rule="[self.findAll('[0-9]+')[1], self.findAll('[0-9]+')[2], self.findAll('[0-9]+')[3]].all(x, int(x) <= 255 && (x == '0' || !x.startsWith('0')))",message="IP address octets must not contain leading zeros, and must be less or equal to 255"
// +optional
InternalMasqueradeSubnet string `json:"internalMasqueradeSubnet"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To avoid issues with Go marshalling this to the empty string, In this case, I think it's safer to add omitempty than to update all validations to account for the empty string.

This would be an exception to our normal guidance for configuration APIs, and I appreciate I told you to remove this earlier. But in this case, because the validation will fail when the empty string is present, we want to ensure that unset Go types don't accidentally add the empty string for this value.

The alternative is to prefix every validation rule with self != '' || which IMO, does make things a little more complicated and then allows users to set the value to the empty string and believe that's a valid value, I would expect we want to prevent users setting the empty string here.

The same is true for the v6 address too.

Adds tests for KEL expressions for internalMasqueradeSubnet fields
Remove redundant ipv4 and ipv6 name for internalMasqueradeSubnet fields
inside their respective structs

Signed-off-by: Ben Pickard <bpickard@redhat.com>
@abhat
Copy link
Contributor

abhat commented Aug 22, 2023

@JoelSpeed thank you for the thorough review. Is this PR good to go from your perspective? Looks like @bpickard22 has addressed most of the comments and has the integration tests in place as well.

cc @trozet

@JoelSpeed
Copy link
Contributor

Yep, I'm good to label this, is the implementation PR ready to merge once this is done? Are we planning to merge during the bug window that's currently in place for 4.14?

@trozet
Copy link
Contributor

trozet commented Aug 23, 2023

@JoelSpeed the implementation is ready. We can merge during the bug window (either way we will need a jira bug) as it is not a risky change to CNO and the OVNK bits have already landed downstream.

@JoelSpeed
Copy link
Contributor

Ack, @bpickard22 Can you link this and the implementation up to a bug so they are tied together, than we can merge if everyone is happy?

@trozet You happy to add LGTM? I'll add approve once that and bug link are done

Copy link
Contributor

@trozet trozet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 24, 2023
@bpickard22
Copy link
Contributor Author

/retitle OCPBUGS-18093:Allow users to specify Gateway Subnet

@openshift-ci openshift-ci bot changed the title Allow users to specify Gateway Subnet OCPBUGS-18093:Allow users to specify Gateway Subnet Aug 24, 2023
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Aug 24, 2023
@openshift-ci-robot
Copy link

@bpickard22: This pull request references Jira Issue OCPBUGS-18093, which is invalid:

  • expected the bug to target the "4.14.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

We currently reserve a range of addresses to configure host to service traffic internally in ovn. We need to allow users to specify this range to avoid conflicting with addresses they use in their local infra

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Aug 24, 2023
@bpickard22
Copy link
Contributor Author

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Aug 24, 2023
@openshift-ci-robot
Copy link

@bpickard22: This pull request references Jira Issue OCPBUGS-18093, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.14.0) matches configured target version for branch (4.14.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @anuragthehatter

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot requested a review from anuragthehatter August 24, 2023 18:50
@JoelSpeed
Copy link
Contributor

/lgtm

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 25, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bpickard22, JoelSpeed, trozet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 25, 2023
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 25, 2023

@bpickard22: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-merge-robot openshift-merge-robot merged commit f9d67cd into openshift:master Aug 25, 2023
@openshift-ci-robot
Copy link

@bpickard22: Jira Issue OCPBUGS-18093: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-18093 has been moved to the MODIFIED state.

In response to this:

We currently reserve a range of addresses to configure host to service traffic internally in ovn. We need to allow users to specify this range to avoid conflicting with addresses they use in their local infra

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jstuever added a commit to jstuever/api that referenced this pull request Sep 13, 2023
…et-api"

This reverts commit f9d67cd, reversing
changes made to 5fdd751.
jstuever added a commit to jstuever/api that referenced this pull request Sep 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants