Upgrade with VMSS enabled cluster only upgrades master nodes, but not agent pools

[croeck]

Is this a request for help?:

YES

Is this an ISSUE or FEATURE REQUEST? (choose one):

ISSUE

What version of acs-engine?:

ersion: canary
GitCommit: b7364fe
GitTreeState: clean

---

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)

Kubernetes 1.10.2

What happened:

Cluster deployed with:

{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorVersion": "1.10.2",
      "kubernetesConfig": {
        "enableDataEncryptionAtRest": false,
        "useManagedIdentity": true,
        "enableRbac": true,
        "networkPolicy": "calico",
        "privateCluster": {
            "enabled": true
        },
        "dnsServiceIP": "10.201.96.96",
        "serviceCidr": "10.201.64.0/18",
        "clusterSubnet": "10.201.128.0/17",
        "etcdDiskSizeGB": "256",
        "dockerEngineVersion": "17.05.*",
        "etcdVersion": "3.3.1"
      }
    },
    "aadProfile": {
      "serverAppID": "*******",
      "clientAppID": "*******",
      "tenantID": "*******",
      "adminGroupID": "*******"
    },
    "masterProfile": {
      "count": 1,
      "dnsPrefix": "*******",
      "vmSize": "Standard_A4_v2",
      "vnetSubnetId": "/subscriptions/*******/resourceGroups/*******/providers/Microsoft.Network/virtualNetworks/*******/subnets/*******",
      "firstConsecutiveStaticIP": "10.201.24.10",
      "vnetCidr": "10.201.0.0/16",
      "distro": "ubuntu"
    },
    "agentPoolProfiles": [
      {
        "name": "agent",
        "count": 1,
        "vmSize": "Standard_A4_v2",
        "vnetSubnetId": "/subscriptions/*******/resourceGroups/*******/providers/Microsoft.Network/virtualNetworks/*******/subnets/*******",
        "availabilityProfile": "VirtualMachineScaleSets",
        "distro": "ubuntu",
        "storageProfile": "ManagedDisks"
      }
    ],
    "linuxProfile": {
      "adminUsername": "*******",
      "ssh": {
        "publicKeys": [
          {
            "keyData": "ssh-rsa *******"
          }
        ]
      }
    }
  }
}

Then an upgrade was attempted:

acs-engine upgrade \
  --subscription-id ******* \
  --deployment-dir *******/_output/*******/ \
  --location westeurope \
  --resource-group ******* \
  --upgrade-version 1.10.3 \
  --debug
INFO[0000] validating...
DEBU[0000] Resolving tenantID for subscriptionID: *******
DEBU[0002] Attempting to load token from cache. path="/Users/*******/.acsengine/cache/536416ce-803d-466d-b19e-05a16d139471_76e0feec-6b7f-41f0-81a7-b1b944520261.token.json"
DEBU[0005] Saved token to cache. path="/Users/*******/.acsengine/cache/536416ce-803d-466d-b19e-05a16d139471_76e0feec-6b7f-41f0-81a7-b1b944520261.token.json"
DEBU[0009] Already registered for "Microsoft.Compute"
DEBU[0009] Already registered for "Microsoft.Storage"
DEBU[0009] Already registered for "Microsoft.Network"
INFO[0011] Name suffix: 87921648
INFO[0011] Gathering agent pool names...
INFO[0011] Master VM name: k8s-master-87921648-0, orchestrator: Kubernetes:1.10.3 (UpgradedMasterVMs)

INFO[0011] Upgrading to Kubernetes version 1.10.3

INFO[0011] Master nodes StorageProfile: ManagedDisks
INFO[0011] Prepping master nodes for upgrade...
INFO[0011] Resource count before running NormalizeResourcesForK8sMasterUpgrade: 10
INFO[0011] Evaluating if agent pool: master, resource: [concat(variables('masterVMNamePrefix'), copyIndex(variables('masterOffset')))] needs to be removed
INFO[0011] Evaluating if extension: [concat(variables('masterVMNamePrefix'), copyIndex(), '/ManagedIdentityExtension')] needs to be removed
INFO[0011] Evaluating if extension: [concat(variables('masterVMNamePrefix'), copyIndex(variables('masterOffset')),'/cse', '-master-', copyIndex(variables('masterOffset')))] needs to be removed
INFO[0011] Resource count after running NormalizeResourcesForK8sMasterUpgrade: 10
INFO[0011] Total expected master count: 1
INFO[0011] Master nodes that need to be upgraded: 0
INFO[0011] Master nodes that have been upgraded: 1
INFO[0011] Starting upgrade of master nodes...
INFO[0011] masterNodesInCluster: 1
INFO[0011] Master VM: k8s-master-87921648-0 is upgraded to expected orchestrator version
INFO[0011] Expected master count: 1, Creating 0 more master VMs
INFO[0011] Cluster upgraded successfully to Kubernetes version 1.10.3

DEBU[0011] output: wrote *******/_output/*******/apimodel.json

The master node was upgrade to 1.10.3, but the agent node remained at 1.10.2. A repeated execution did not solve the Problem.

What you expected to happen:

The node to also be upgraded to 1.10.3

How to reproduce it (as minimally and precisely as possible):

Deploy cluster using VMSS and Kubernetes 1.10.2, then try to upgrade to 1.10.3

Anything else we need to know:

I suspect the issue starts here:

func (uc *UpgradeCluster) getClusterNodeStatus(subscriptionID uuid.UUID, resourceGroup string) error {
	vmListResult, err := uc.Client.ListVirtualMachines(resourceGroup)
...

This function only retrieves the VMs of AvailabilitySets, but not VMSS.

[khenidak]

@sozercan FYI