Kubernetes Hybrid Cluster with Custom Vnet - Windows node issue

[mboret]

Is this a request for help?:
No

Is this an ISSUE or FEATURE REQUEST? (choose one):
ISSUE

What version of acs-engine?: v0.15

---

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
Kubernetes

What happened:
I'm not able to create a new hybrid cluster with a custom Vnet with Azure CNI or Kubenet.
In both case the error is:

The template variable 'subnet' is not found

This appears during the Windows node creation.

What you expected to happen:
Be able to have a working kubernetes hybrid cluster with a custom Vnet.

How to reproduce it (as minimally and precisely as possible):

acs-engine: v0.15
Template:

{
    "apiVersion": "vlabs",
    "properties": {
      "orchestratorProfile": {
        "orchestratorType": "Kubernetes",
        "orchestratorRelease": "1.10",
        "kubernetesConfig": {
          "networkPolicy": "azure", # or "none"
          "enableRbac": true
        }
      },
      "masterProfile": {
        "count": 2,
        "dnsPrefix": "DNSPREFIX",
        "vmSize": "Standard_D2_v2",
        "vnetSubnetId": "/subscriptions/SUBSCRIPTION/resourceGroups/RESOURCEGROUP/providers/Microsoft.Network/virtualNetworks/VNETNAME/subnets/SUBNETNAME",
        "firstConsecutiveStaticIP": "10.198.3.239",
        "OSDiskSizeGB": 50
      },
      "agentPoolProfiles": [
        {
          "name": "linuxpool",
          "count": 2,
          "vmSize": "Standard_D2_v2",
          "availabilityProfile": "AvailabilitySet",
          "OSDiskSizeGB": 50,
          "distro": "ubuntu",
          "vnetSubnetId": "/subscriptions/SUBSCRIPTION/resourceGroups/RESOURCEGROUP/providers/Microsoft.Network/virtualNetworks/VNETNAME/subnets/SUBNETNAME"          
        },
        {
          "name": "windowspool",
          "count": 2,
          "vmSize": "Standard_D2_v3",
          "availabilityProfile": "AvailabilitySet",
          "osType": "Windows",
          "OSDiskSizeGB": 100,
          "vnetSubnetId": "/subscriptions/SUBSCRIPTION/resourceGroups/RESOURCEGROUP/providers/Microsoft.Network/virtualNetworks/VNETNAME/subnets/SUBNETNAME"                
        }
      ],
      "windowsProfile": {
        "adminUsername": "ADMINUSERWINDOWS",
        "adminPassword": "WINDOWSPASSWORD"
      },
      "linuxProfile": {
        "adminUsername": "ADMINUSERLINUX",
        "ssh": {
          "publicKeys": [
            {
              "keyData": "LINUXPUBLICKEY"
            }
          ]
        }
      },
      "servicePrincipalProfile": {
        "clientId": "SPNAME",
        "secret": "SPPASSWORD"
      }
    }
  }

Anything else we need to know:

This issue #2565 and the proposed fix(Rename: variables('subnet') by variables('vnetSubnetID')):

With Kubenet:

I'm able to deploy a cluster with all nodes(Windows and Linux) with the READY state but:

1. The Windows node state has no role(23774k8s9010 Ready < none > 1h v1.10.0 < none > Windows Server Datacenter 10.0.16299.309 docker://17.6.2 )

2. The node resource metrics is not working(0% Cpu/0% Memory)

3. I cannot deploy a pod on the windows node. The pod is blocked with the status "Container Creating"

With Azure CNI, the error during the cluster creation is:

Step: k8s-master-232424-0/cse0 Status: Conflict

The resource operation completed with terminal provisioning state 'Failed'.details: code:VMExtensionProvisioningError,message:VM has reported a failure when processing extension 'cse0'. Error message:"Enable failed: failed to execute command: command terminated with exit status=3[stdout][stderr]"

Currently I'm able to deploy and have a working(with some non blocking issues) hybrid cluster with custom Vnet only by using this PR #1810 with kubenet and k8s 1.8

In summary, the issue seems only related to the Windows node with custom VNET.
And as with the latest acs-engine release we are not receiving the message "validation error if custom VNET + Windows" (#2168) The creation should be working as expected.

[mvelazquezm]

I have exactly the same Issue. I´ve tried with ACS Engine 0.12, 0.13, 0.14 and 0.15 And K8S 1.9.3.

I also have same issue (With Azure CNI, the error during the cluster creation) with Linux only cluster, and Azure CNI over custom VNet.

{ "apiVersion": "vlabs", "properties": { "orchestratorProfile": { "orchestratorType": "Kubernetes", "orchestratorRelease": "1.9", "orchestratorVersion": "1.9.3", "kubernetesConfig": { "networkPolicy": "azure", "privateCluster": { "enabled": true } } }, "masterProfile": { "count": 1, "dnsPrefix": "app-k8s-linux-cluster", "vmSize": "Standard_D2_v2", "vnetSubnetId": "/subscriptions/xxxx/resourceGroups/GR-NETWORKING-MVM/providers/Microsoft.Network/virtualNetworks/VNET-MVM/subnets/SUBNET-K8SL", "firstConsecutiveStaticIP": "10.116.5.102" }, "agentPoolProfiles": [ { "name": "linuxpool1", "count": 2, "vmSize": "Standard_D2_v2", "vnetSubnetId": "/subscriptions/xxxxx/resourceGroups/GR-NETWORKING-MVM/providers/Microsoft.Network/virtualNetworks/VNET-MVM/subnets/SUBNET-K8SL", "availabilityProfile": "AvailabilitySet" } ], "linuxProfile": { "adminUsername": "xxxxx", "ssh": { "publicKeys": [ { "keyData": "xxxx" } ] } }, "servicePrincipalProfile": { "clientId": "xxxx", "secret": "xxxx" } } }

Thanks

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contribution. Note that acs-engine is deprecated--see https://github.com/Azure/aks-engine instead.