Dual stack follow-ups #2173

aojea · 2021-04-01T12:50:29Z

Fix documentation and workaround 1.20 versions to enable the feature gate

This needs an image bump for having the kindnet dualstack versions, so pods can get dual-stack addresses

aojea · 2021-04-01T12:51:13Z

/hold
it needs images bumps

aojea · 2021-04-01T12:54:14Z

pkg/cluster/internal/create/create.go

+  IPv6DualStack: true
+`
+	if len(cfg.KubeadmConfigPatches) > 0 {
+		cfg.KubeadmConfigPatches = append(cfg.KubeadmConfigPatches, dualPatch)


is this ok, or should we use Json6902?

There's a much easier way to default this when we generate the base kind config which also respects the user patch applied on top. See how we handled it for the PV feature gatr

aojea · 2021-04-01T12:55:44Z

pkg/cluster/internal/create/create.go

+	}
+
+	// for 1.20 we have to explicitly enable the feature gate
+	// we can patch kubeadm here so we avoid to plumb this hack further


kubeadm adds the corresponding feature gates flags to the other components, controller-manager, apiserver, kube-proxy ... and validates that podsubnet and serviceSubnet are correct ... so we don't need to set the feature flags

aojea · 2021-04-01T12:57:21Z

/assign @BenTheElder
/cc @aspenmesh @howardjohn

k8s-ci-robot · 2021-04-01T12:57:23Z

@aojea: GitHub didn't allow me to request PR reviews from the following users: aspenmesh, howardjohn.

Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/assign @BenTheElder
/cc @aspenmesh @howardjohn

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

aojea · 2021-04-01T14:17:23Z

after updated the kindnet image

kubectl set image ds/kindnet kindnet-cni=kindest/kindnetd:v20210326-1e038dc5 -n kube-system

and delete the old kindnet pods that should be crashlooping because it can't handle dualstack correctly

I0401 13:49:37.797835       1 main.go:185] handling current node
panic: Can't synchronize rules after 3 attempts: running [/usr/sbin/iptables -t nat -C KIND-MASQ-AGENT -d 10.244.0.0/16,fd00:10:244::/56 -j RETURN -m comment --comment kind-masq-agent: local traffic is not subject to MASQUERADE --wait]: exit status 2: iptables v1.8.5 (legacy): invalid mask `56' specified
Try `iptables -h' or 'iptables --help' for more information.

I have some problems because the apiserver fails to validate everything after this error

E0401 12:44:27.426813       1 controller.go:152] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/172.18.0.2, ResourceVersion: 0, AdditionalErrorMsg: 
E0401 12:44:27.435678       1 fieldmanager.go:186] [SHOULD NOT HAPPEN] failed to update managedFields for /, Kind=: failed to convert new object (/v1, Kind=Node) to smd typed: .status.addresses: duplicate entries for key [type="InternalIP"]

@liggitt can you give me some hints to debug this further?

EDIT

this is with 1.20.2

aojea · 2021-04-01T16:28:33Z

/retest

liggitt · 2021-04-01T16:35:20Z

E0401 12:44:27.435678       1 fieldmanager.go:186] [SHOULD NOT HAPPEN] failed to update managedFields for /, Kind=: failed to convert new object (/v1, Kind=Node) to smd typed: .status.addresses: duplicate entries for key [type="InternalIP"]

@liggitt can you give me some hints to debug this further?

sounds like kubernetes/kubernetes#88182

cc @apelisse

liggitt · 2021-04-01T16:37:10Z

duplicate entries for key [type="InternalIP"]

Looks like a retread of kubernetes/kubernetes#79391 (comment), but now with server-side apply.

liggitt · 2021-04-01T16:44:50Z

If we can have existing objects with duplicate items here (and it sounds like we can), the managedFields update cannot break update of those objects. Complaining on Apply requests is in-bounds (though it sounds like the merge key selected for the field may be wrong in the first place if we expect to have multiple entries of the same address type with different IP families)

apelisse · 2021-04-01T16:45:42Z

Yeah, I'd be very surprised if it does, I think that's a red-herring.

BenTheElder · 2021-04-01T17:09:51Z

site/content/docs/user/configuration.md

+{{< /codeFromInline >}}
+
+##### Dual Stack clusters
+You can run dual stack clusters using `kind`, on kubernetes versions 1.20+.


Please note that this requires 0.11.0+

aojea · 2021-04-01T18:35:46Z

@liggitt @apelisse indeed it was a redherring, thanks for your quick response, I can't explain why by it was caused by a wrong kubeadm configuration, there was also rbac errors on the scheduler and the controller manager.

@BenTheElder PTAL, it needs the new kindnet image in order to work

kubectl set image ds/kindnet kindnet-cni=kindest/kindnetd:v20210326-1e038dc5 -n kube-system

$ kubectl apply -f https://gist.githubusercontent.com/aojea/90768935ab71cb31950b6a13078a7e92/raw/99ceac308f2b2658c7313198a39fbe24b155ae68/dual-stack.yaml
$ kubectl get endpointslices -o wide
NAME                            ADDRESSTYPE   PORTS   ENDPOINTS                       AGE
kubernetes                      IPv4          6443    172.18.0.2                      4m24s
my-service-prefer-dual-jswjr    IPv4          80      10.244.0.6,10.244.0.5           45s
my-service-prefer-dual-vvkdt    IPv6          80      fd00:10:244::3,fd00:10:244::2   45s
my-service-require-dual-4hqq2   IPv6          80      fd00:10:244::3,fd00:10:244::2   45s
my-service-require-dual-75hk8   IPv4          80      10.244.0.6,10.244.0.5           45s
my-service-v4-429wd             IPv4          80      10.244.0.6,10.244.0.5           45s
my-service-v6-crtqs             IPv6          80      fd00:10:244::3,fd00:10:244::2   45s

aojea · 2021-04-01T18:37:59Z

/hold cancel

BenTheElder · 2021-04-01T19:00:30Z

pkg/cluster/internal/kubeadm/config.go

+	// dual-stack is only supported in 1.20+
+	// TODO: remove this when 1.20 is EOL or we no longer support
+	// dual-stack for 1.20 in KIND
+	if ver.LessThan(version.MustParseSemantic("v1.21.0")) &&


we should really use the version for the PR that enabled it by default so it works with arbitrary commits 🙃

if you checkout kubernetes master then the commit you can grab the version.

can be a follow-up

this was the kubeadm one kubernetes/kubernetes#99294

and this the kubernetes one kubernetes/kubernetes#98969

the must parse semantic doesn't accept versions with letters IIRC

BenTheElder · 2021-04-01T19:02:14Z

@aojea upgrade the kindnetd image / manifest? you have access 🙃

BenTheElder · 2021-04-01T19:02:27Z

/lgtm
/approve
can be handled in another PR, along with refining the versions

k8s-ci-robot · 2021-04-01T19:02:36Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aojea, BenTheElder

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [BenTheElder,aojea]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

fix dualstack docs

114dd68

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Apr 1, 2021

k8s-ci-robot requested review from krzyzacy and munnerz April 1, 2021 12:50

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Apr 1, 2021

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 1, 2021

aojea force-pushed the dual_followup branch from 5bb7738 to ed615f0 Compare April 1, 2021 12:52

aojea commented Apr 1, 2021

View reviewed changes

k8s-ci-robot assigned BenTheElder Apr 1, 2021

aojea force-pushed the dual_followup branch 2 times, most recently from b24f7c0 to fc07f05 Compare April 1, 2021 13:46

BenTheElder reviewed Apr 1, 2021

View reviewed changes

aojea added 2 commits April 1, 2021 20:32

reorganize IPFamily docs

2a91063

enable kubeadm dual-stack FG for 1.20

31e5956

aojea force-pushed the dual_followup branch from cc4fa9c to 31e5956 Compare April 1, 2021 18:32

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 1, 2021

BenTheElder reviewed Apr 1, 2021

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 1, 2021

k8s-ci-robot merged commit bbede40 into kubernetes-sigs:main Apr 1, 2021

aojea mentioned this pull request Apr 13, 2021

Change node.status.addresses to "atomic" (Dual-stack; log errors for multiple InternalIP items in Node objects) kubernetes/kubernetes#101029

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dual stack follow-ups #2173

Dual stack follow-ups #2173

aojea commented Apr 1, 2021 •

edited

Loading

aojea commented Apr 1, 2021

aojea Apr 1, 2021

BenTheElder Apr 1, 2021

aojea Apr 1, 2021

aojea commented Apr 1, 2021

k8s-ci-robot commented Apr 1, 2021

aojea commented Apr 1, 2021 •

edited

Loading

aojea commented Apr 1, 2021

liggitt commented Apr 1, 2021

liggitt commented Apr 1, 2021 •

edited

Loading

liggitt commented Apr 1, 2021

apelisse commented Apr 1, 2021

BenTheElder Apr 1, 2021

aojea commented Apr 1, 2021

aojea commented Apr 1, 2021

BenTheElder Apr 1, 2021

BenTheElder Apr 1, 2021

aojea Apr 1, 2021

aojea Apr 1, 2021

BenTheElder commented Apr 1, 2021

BenTheElder commented Apr 1, 2021

k8s-ci-robot commented Apr 1, 2021

Dual stack follow-ups #2173

Dual stack follow-ups #2173

Conversation

aojea commented Apr 1, 2021 • edited Loading

aojea commented Apr 1, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aojea commented Apr 1, 2021

k8s-ci-robot commented Apr 1, 2021

aojea commented Apr 1, 2021 • edited Loading

aojea commented Apr 1, 2021

liggitt commented Apr 1, 2021

liggitt commented Apr 1, 2021 • edited Loading

liggitt commented Apr 1, 2021

apelisse commented Apr 1, 2021

Choose a reason for hiding this comment

aojea commented Apr 1, 2021

aojea commented Apr 1, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenTheElder commented Apr 1, 2021

BenTheElder commented Apr 1, 2021

k8s-ci-robot commented Apr 1, 2021

aojea commented Apr 1, 2021 •

edited

Loading

aojea commented Apr 1, 2021 •

edited

Loading

liggitt commented Apr 1, 2021 •

edited

Loading