Availability zones duplicated when generating worker machineset assets

[ingvagabund]

Version

$ openshift-install version
<your output here>

Platform (aws|libvirt|openstack):

aws

What happened?

When running our CI over cluster-autoscaler (on top of a cluster created by the installer) we sometimes see only two machinesets created instead of three. The logs shows machineset in us-east-1a AZ has two instances instead of one. After checking the code responsible for generating machineset manifests [1] we see the number of replicas for a machineset in each AZ equals the number of occurrences of the same AZ in the list of all AZs passed to MachineSets() function. The list of available AZs is read from a pool passed down the invocation [2]. The list of AZs in the pool is set by reading ic.Platform.AWS.DefaultMachinePlatform and pool.Platform.AWS variables [3]. Which are set by dependencies.Get(clusterID, installconfig, rhcosImage, wign) line [4]. Something in the code is duplicating us-east-1a AZ which results in two machine sets generated instead of three.

I0502 06:43:51.907548    4708 framework.go:397] >>> kubeConfig: /tmp/admin.kubeconfig
I0502 06:43:52.773856    4708 utils.go:89] Cluster size is 6 nodes
I0502 06:43:52.773882    4708 utils.go:240] Cluster size expected to be 6 nodes
I0502 06:43:52.790211    4708 utils.go:101] MachineSet "ci-op-16dt3s03-1227b-4mhd8-worker-us-east-1a" replicas 2. Ready: 2, available 2
I0502 06:43:52.790245    4708 utils.go:101] MachineSet "ci-op-16dt3s03-1227b-4mhd8-worker-us-east-1b" replicas 1. Ready: 1, available 1

[1]

installer/pkg/asset/machines/aws/machinesets.go

Line 37 in 30e4f71

replicas++

[2]

installer/pkg/asset/machines/worker.go

Line 146 in 30e4f71

sets, err := aws.MachineSets(clusterID.InfraID, ic, &pool, string(*rhcosImage), "worker", "worker-user-data")

[3]

installer/pkg/asset/machines/worker.go

Lines 136 to 137 in 30e4f71

	mpool.Set(ic.Platform.AWS.DefaultMachinePlatform)
	mpool.Set(pool.Platform.AWS)

[4]

installer/pkg/asset/machines/worker.go

Line 118 in 30e4f71

dependencies.Get(clusterID, installconfig, rhcosImage, wign)

What you expected to happen?

One machineset created for each AZ with exactly one replica

How to reproduce it (as minimally and precisely as possible)?

Observed in CI: https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_kubernetes-autoscaler/90/pull-ci-openshift-kubernetes-autoscaler-master-e2e-aws-operator/270/

Search for:

• [SLOW TEST:244.362 seconds]
[Feature:Machines] Managed cluster should
/go/src/k8s.io/autoscaler/test/openshift/vendor/github.com/openshift/cluster-api-actuator-pkg/pkg/e2e/infra/infra.go:127
  recover from deleted worker machines
  /go/src/k8s.io/autoscaler/test/openshift/vendor/github.com/openshift/cluster-api-actuator-pkg/pkg/e2e/infra/infra.go:214
------------------------------
[Feature:Machines] Managed cluster should 
  grow or decrease when scaling out or in
  /go/src/k8s.io/autoscaler/test/openshift/vendor/github.com/openshift/cluster-api-actuator-pkg/pkg/e2e/infra/infra.go:261
STEP: checking initial cluster state
I0502 06:43:51.907548    4708 framework.go:397] >>> kubeConfig: /tmp/admin.kubeconfig
I0502 06:43:52.773856    4708 utils.go:89] Cluster size is 6 nodes
I0502 06:43:52.773882    4708 utils.go:240] Cluster size expected to be 6 nodes
I0502 06:43:52.790211    4708 utils.go:101] MachineSet "ci-op-16dt3s03-1227b-4mhd8-worker-us-east-1a" replicas 2. Ready: 2, available 2
I0502 06:43:52.790245    4708 utils.go:101] MachineSet "ci-op-16dt3s03-1227b-4mhd8-worker-us-east-1b" replicas 1. Ready: 1, available 1

$ your-commands-here

Anything else we need to know?

Enter text here.

References

• enter text here.

[abhinavdahiya]

We purposefully turned out AZs to 2
openshift/release#3615

So 3 machines her stripped to 2 AZs, which is expected.

/close

[openshift-ci-robot]

@abhinavdahiya: Closing this issue.

In response to this:

We purposefully turned out AZs to 2
openshift/release#3615

So 3 machines her stripped to 2 AZs, which is expected.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.