Unable to upgrade tenant with two pools due to race condition caused by startupProbe in v4.5.2

[astraldawn]

Expected Behavior

1. Update .spec.image in tenant with two pools
2. StatefulSet for each pool restarts with updated image

Current Behavior

1. Update .spec.image in tenant with two pools
2. StatefulSet for each pool stuck with status:

Pod	StatefulSet 0	StatefulSet 1
0	Running	Running
1	Running	Running
2	Running	Running
3	Unavailable (Notready)	Unavailable (Notready)

3. Indicative logs for the unavailable pods:

StatefulSet 0-3	StatefulSet 1-3
lookup tenant-ss-1-3... no such host (*net.DNSError)	lookup tenant-ss-0-3... no such host (*net.DNSError)

Possible Solution

Downgrade to v4.5.1

We considered modifying tenant spec but this is not possible as startupProbe is hardcoded (https://github.com/minio/operator/pull/1284/files#diff-dd388a34c55183c615d3aed4bb2004c13ad10d64c2163370890a0746b00cd707R365)

Steps to Reproduce (for bugs)

See current behavour

Context

Unable to upgrade tenant with 2 pools. Not sure if this impacts a tenant with > 2 pools.

Regression

Yes, operator v4.5.2, due to #1284

Upgrade operation was run on these operator versions no issues:

• v4.4.17
• v4.5.1

Your Environment

• Version used (minio-operator): v4.5.2
• Environment name and version (e.g. kubernetes v1.17.2): v1.19.16

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[cesnietor]

we'll try to reproduce

[astraldawn]

Should be fixed by #1315

[cesnietor]

@astraldawn were you able to test it with latest operator? thanks for responding.

[astraldawn]

Tested with v4.5.8

[cniackz]

Yes, this is a known issue in version v4.5.2 that has been corrected. Thank you for testing, and please continue upgrading the Operator to the latest available version. You can find more information in the documentation here: https://github.com/minio/wiki/wiki/DNSError-in-Operator-version-4.5.2