feat: allow creation of dualstack Windows clusters #4176

vavenk-ms · 2021-01-18T13:24:11Z

Reason for Change:

Allow creation of dual stack Windows Kubernetes clusters. This was restricted as Windows did not support dual stack. From Windows version 2004 (10.0.19041.610) and Kubernetes version 1.19, dual stack is supported. https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#ipv4-ipv6-dual-stack

Issue Fixed:

#4171

Credit Where Due:

Does this change contain code from or inspired by another project?

No
Yes

If "Yes," did you notify that project's maintainers and provide attribution?

No
Yes

Requirements:

uses conventional commit messages
includes documentation
adds unit tests
tested upgrade from previous version

Notes:

vavenk-ms · 2021-01-18T13:42:47Z

Since the Kubernetes config for Windows was slightly different from Linux, created a separate template instead of adding another pool in the same one.
I wasn't able to get all e2e tests for Windows to pass with this template with/without dualstack, I've listed details of failures below. Some of these passed when I ran just the Windows tests.

Summarizing 5 Failures:

[Fail] Azure Container Cluster using the Kubernetes Orchestrator regardless of agent pool type [It] should have functional container networking DNS
/aks-engine/test/e2e/kubernetes/kubernetes_test.go:889

[Fail] Azure Container Cluster using the Kubernetes Orchestrator with a windows agent pool [It] should be able to deploy and scale an iis webserver
/aks-engine/test/e2e/kubernetes/kubernetes_test.go:2013

[Fail] Azure Container Cluster using the Kubernetes Orchestrator with a windows agent pool [It] should be able to resolve DNS across windows and linux deployments
/aks-engine/test/e2e/kubernetes/kubernetes_test.go:2129

[Fail] Azure Container Cluster using the Kubernetes Orchestrator with a windows agent pool [It] should expect containers to be recreated after node restart
/aks-engine/test/e2e/kubernetes/kubernetes_test.go:2354

[Fail] Azure Container Cluster using the Kubernetes Orchestrator with a windows agent pool [It] should be able to rotate docker logs
/aks-engine/test/e2e/kubernetes/kubernetes_test.go:2377

mboersma · 2021-01-19T16:52:24Z

/azp run pr-e2e

azure-pipelines · 2021-01-19T16:52:35Z

Azure Pipelines successfully started running 1 pipeline(s).

codecov · 2021-01-19T17:07:27Z

Codecov Report

Merging #4176 (93a0d8b) into master (b9c9dda) will decrease coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #4176      +/-   ##
==========================================
- Coverage   72.06%   72.04%   -0.03%     
==========================================
  Files         141      141              
  Lines       21686    21636      -50     
==========================================
- Hits        15628    15587      -41     
+ Misses       5107     5099       -8     
+ Partials      951      950       -1

Impacted Files	Coverage Δ
pkg/api/vlabs/validate.go	`81.52% <100.00%> (+0.17%)`	⬆️
pkg/engine/params_k8s.go	`79.57% <0.00%> (-0.56%)`	⬇️
pkg/engine/template_generator.go	`68.20% <0.00%> (-0.15%)`	⬇️
pkg/api/defaults-custom-cloud-profile.go	`85.12% <0.00%> (-0.13%)`	⬇️
pkg/api/addons.go	`98.03% <0.00%> (-0.08%)`	⬇️
pkg/engine/artifacts.go	`98.90% <0.00%> (-0.05%)`	⬇️
pkg/api/convertertoapi.go	`94.01% <0.00%> (-0.02%)`	⬇️
pkg/api/types.go	`92.85% <0.00%> (-0.02%)`	⬇️
pkg/api/converterfromapi.go	`95.69% <0.00%> (-0.01%)`	⬇️
pkg/api/common/versions.go	`96.37% <0.00%> (ø)`
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b9c9dda...93a0d8b. Read the comment docs.

pkg/api/vlabs/validate_test.go

Co-authored-by: James Sturtevant <jsturtevant@gmail.com>

examples/dualstack/README.md

pkg/api/vlabs/validate.go

jsturtevant · 2021-01-20T19:31:08Z

Do you have a handle on why the tests are failing and what is needed to pass? Have you tried this with Containerd?

vavenk-ms · 2021-01-21T11:01:30Z

Do you have a handle on why the tests are failing and what is needed to pass? Have you tried this with Containerd?

I haven't dug into it, it looks like something to do with azurecni, pods are not able to come up and I see "No available addresses". However, these tests pass when I run them in isolation, so a mix of some of these tests are not working. In addition, I didn't notice any issues while running our workloads, in similar dual stack clusters. And, no I haven't tested this with containerd.

vavenk-ms · 2021-01-27T10:06:14Z

@jsturtevant, looks like the tests were failing because there was an issue with azure cni (Azure/azure-container-networking#760) which was fixed recently (Azure/azure-container-networking#755) and updated in aks-engine (#4183). With the latest aks-engine all the e2e tests pass with both docker and containerd runtimes.

junit-containerd.txt
junit-docker.txt

jsturtevant · 2021-01-29T18:05:19Z

/azp run pr-e2e

azure-pipelines · 2021-01-29T18:05:29Z

Azure Pipelines successfully started running 1 pipeline(s).

…enk/allowWinDualStack

jsturtevant · 2021-02-03T16:08:40Z

/azp run pr-e2e

azure-pipelines · 2021-02-03T16:08:50Z

Azure Pipelines successfully started running 1 pipeline(s).

…enk/allowWinDualStack

jsturtevant · 2021-02-22T16:56:33Z

/azp run pr-e2e

azure-pipelines · 2021-02-22T16:56:45Z

Azure Pipelines successfully started running 1 pipeline(s).

jsturtevant · 2021-02-22T17:11:16Z

/azp run pr-e2e

azure-pipelines · 2021-02-22T17:11:26Z

Azure Pipelines successfully started running 1 pipeline(s).

jsturtevant · 2021-02-24T16:37:29Z

/lgtm

this will require an update to the provisioning scripts: https://github.com/Azure/aks-engine/blob/master/docs/topics/windows-provisioning-scripts.md

@aramase could you give a look on more look from ipv6 perspective?

aramase · 2021-02-24T17:30:27Z

staging/provisioning/windows/windowsnodereset.ps1

@@ -15,6 +15,7 @@ $global:NetworkMode = "L2Bridge"
 $global:NetworkPlugin = $Global:ClusterConfiguration.Cni.Name
 $global:ContainerRuntime = $Global:ClusterConfiguration.Cri.Name
 $UseContainerD = ($global:ContainerRuntime -eq "containerd")
+$IsDualStackEnabled = $Global:ClusterConfiguration.Kubernetes.Kubeproxy.FeatureGates -contains "IPv6DualStack=true"


Not a blocker for this PR but we should also think about how we would do this in v1.21+. Dual-stack is beta as of v1.21 and these feature gates are enabled by default. So they will not be explicitly set.

aramase

/lgtm

acs-bot · 2021-02-24T17:35:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aramase, jsturtevant, vavenk-ms

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jsturtevant]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

vavenk-ms · 2021-02-25T16:27:15Z

@jsturtevant

$ kubectl get nodes -o wide
NAME                    STATUS   ROLES    AGE     VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                    KERNEL-VERSION     CONTAINER-RUNTIME
6837k8s00000000         Ready    agent    5m37s   v1.19.7   10.240.0.34    <none>        Windows Server Datacenter   10.0.19041.804     docker://19.3.14
k8s-master-68377600-0   Ready    master   9m23s   v1.19.7   10.255.255.5   <none>        Ubuntu 18.04.5 LTS          5.4.0-1040-azure   docker://19.3.14

$ kubectl describe node 6837k8s00000000
Name:               6837k8s00000000
Roles:              agent
Labels:             agentpool=windowspool2
                    beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=Standard_D2_v3
                    beta.kubernetes.io/os=windows
                    failure-domain.beta.kubernetes.io/region=eastus
                    failure-domain.beta.kubernetes.io/zone=0
                    kubernetes.azure.com/cluster=kubernetes-eastus-49399
                    kubernetes.azure.com/role=agent
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=6837k8s00000000
                    kubernetes.io/os=windows
                    kubernetes.io/role=agent
                    node-role.kubernetes.io/agent=
                    node.kubernetes.io/instance-type=Standard_D2_v3
                    node.kubernetes.io/windows-build=10.0.19041
                    storageprofile=managed
                    storagetier=Standard_LRS
                    topology.kubernetes.io/region=eastus
                    topology.kubernetes.io/zone=0
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 25 Feb 2021 16:07:19 +0000
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  6837k8s00000000
  AcquireTime:     <unset>
  RenewTime:       Thu, 25 Feb 2021 16:16:00 +0000
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Thu, 25 Feb 2021 16:07:36 +0000   Thu, 25 Feb 2021 16:07:36 +0000   RouteCreated                 RouteController created a route
  MemoryPressure       False   Thu, 25 Feb 2021 16:12:33 +0000   Thu, 25 Feb 2021 16:07:19 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Thu, 25 Feb 2021 16:12:33 +0000   Thu, 25 Feb 2021 16:07:19 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Thu, 25 Feb 2021 16:12:33 +0000   Thu, 25 Feb 2021 16:07:19 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Thu, 25 Feb 2021 16:12:33 +0000   Thu, 25 Feb 2021 16:07:29 +0000   KubeletReady                 kubelet is posting ready status
Addresses:
  Hostname:    6837k8s00000000
  InternalIP:  10.240.0.34
  InternalIP:  2001:1234:5678:9abc::5
Capacity:
  attachable-volumes-azure-disk:  4
  cpu:                            2
  ephemeral-storage:              133703676Ki
  memory:                         8388148Ki
  pods:                           30
Allocatable:
  attachable-volumes-azure-disk:  4
  cpu:                            2
  ephemeral-storage:              133703676Ki
  memory:                         6290996Ki
  pods:                           30
System Info:
  Machine ID:                 6837k8s00000000
  System UUID:                00000000-0000-0000-0000-00000000000
  Boot ID:
  Kernel Version:             10.0.19041.804
  OS Image:                   Windows Server Datacenter
  Operating System:           windows
  Architecture:               amd64
  Container Runtime Version:  docker://19.3.14
  Kubelet Version:            v1.19.7
  Kube-Proxy Version:         v1.19.7
PodCIDR:                      fc00:0:0:1::/64
PodCIDRs:                     fc00:0:0:1::/64
ProviderID:                   azure:///subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/kubernetes-eastus-49399/providers/Microsoft.Compute/virtualMachineScaleSets/6837k8s00/virtualMachines/0
Non-terminated Pods:          (0 in total)
  Namespace                   Name    CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                   ----    ------------  ----------  ---------------  -------------  ---
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                       Requests  Limits
  --------                       --------  ------
  cpu                            0 (0%)    0 (0%)
  memory                         0 (0%)    0 (0%)
  ephemeral-storage              0 (0%)    0 (0%)
  attachable-volumes-azure-disk  0         0
Events:
  Type    Reason                   Age                    From     Message
  ----    ------                   ----                   ----     -------
  Normal  Starting                 8m43s                  kubelet  Starting kubelet.
  Normal  NodeHasSufficientMemory  8m43s (x2 over 8m43s)  kubelet  Node 6837k8s00000000 status is now: NodeHasNoDiskPressure
  Normal  NodeReady                8m33s                  kubelet  Node 6837k8s00000000 status is now: NodeReady

feat: allow creation of dualstack Windows clusters

c0ef5f5

mboersma added the windows label Jan 19, 2021

mboersma requested review from aramase, jsturtevant and marosset January 19, 2021 16:52

jsturtevant reviewed Jan 19, 2021

View reviewed changes

pkg/api/vlabs/validate_test.go Outdated Show resolved Hide resolved

jsturtevant reviewed Jan 19, 2021

View reviewed changes

pkg/api/vlabs/validate_test.go Outdated Show resolved Hide resolved

Apply suggestions from code review

fe1ae00

Co-authored-by: James Sturtevant <jsturtevant@gmail.com>

aramase reviewed Jan 20, 2021

View reviewed changes

examples/dualstack/README.md Outdated Show resolved Hide resolved

pkg/api/vlabs/validate.go Outdated Show resolved Hide resolved

vavenk-ms requested review from jsturtevant and aramase January 27, 2021 10:07

Merge branch 'master' into vavenk/allowWinDualStack

b698dff

vavenk added 2 commits February 3, 2021 18:29

Merge branch 'master' of https://github.com/Azure/aks-engine into vav…

df286b2

…enk/allowWinDualStack

Disable IPv6-only on Windows

d991f32

Merge branch 'master' of https://github.com/Azure/aks-engine into vav…

8adf9e9

…enk/allowWinDualStack

jsturtevant mentioned this pull request Feb 22, 2021

fix: restart hns for Windows dual-stack #4281

Closed

8 tasks

fix: restart hns for Windows dual-stack

93a0d8b

aramase reviewed Feb 24, 2021

View reviewed changes

jsturtevant merged commit 9047a45 into Azure:master Mar 2, 2021

gofish-bot mentioned this pull request Mar 15, 2021

aks-engine 0.61.0 fishworks/fish-food#1279

Merged

bridgetkromhout mentioned this pull request Feb 11, 2022

Allow creation of dual stack Windows clusters #4171

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: allow creation of dualstack Windows clusters #4176

feat: allow creation of dualstack Windows clusters #4176

vavenk-ms commented Jan 18, 2021

vavenk-ms commented Jan 18, 2021

mboersma commented Jan 19, 2021

azure-pipelines bot commented Jan 19, 2021

codecov bot commented Jan 19, 2021 •

edited

Loading

jsturtevant commented Jan 20, 2021

vavenk-ms commented Jan 21, 2021

vavenk-ms commented Jan 27, 2021 •

edited

Loading

jsturtevant commented Jan 29, 2021

azure-pipelines bot commented Jan 29, 2021

jsturtevant commented Feb 3, 2021

azure-pipelines bot commented Feb 3, 2021

jsturtevant commented Feb 22, 2021

azure-pipelines bot commented Feb 22, 2021

jsturtevant commented Feb 22, 2021

azure-pipelines bot commented Feb 22, 2021

jsturtevant commented Feb 24, 2021

aramase Feb 24, 2021

aramase left a comment

acs-bot commented Feb 24, 2021

vavenk-ms commented Feb 25, 2021

feat: allow creation of dualstack Windows clusters #4176

feat: allow creation of dualstack Windows clusters #4176

Conversation

vavenk-ms commented Jan 18, 2021

vavenk-ms commented Jan 18, 2021

mboersma commented Jan 19, 2021

azure-pipelines bot commented Jan 19, 2021

codecov bot commented Jan 19, 2021 • edited Loading

Codecov Report

jsturtevant commented Jan 20, 2021

vavenk-ms commented Jan 21, 2021

vavenk-ms commented Jan 27, 2021 • edited Loading

jsturtevant commented Jan 29, 2021

azure-pipelines bot commented Jan 29, 2021

jsturtevant commented Feb 3, 2021

azure-pipelines bot commented Feb 3, 2021

jsturtevant commented Feb 22, 2021

azure-pipelines bot commented Feb 22, 2021

jsturtevant commented Feb 22, 2021

azure-pipelines bot commented Feb 22, 2021

jsturtevant commented Feb 24, 2021

aramase Feb 24, 2021

Choose a reason for hiding this comment

aramase left a comment

Choose a reason for hiding this comment

acs-bot commented Feb 24, 2021

vavenk-ms commented Feb 25, 2021

codecov bot commented Jan 19, 2021 •

edited

Loading

vavenk-ms commented Jan 27, 2021 •

edited

Loading