fix(controller): fix race condition in updating ephemeral metadata

[y-rabie]

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this is a chore.
• The title of the PR is (a) conventional with a list of types and scopes found here, (b) states what changed, and (c) suffixes the related issues number. E.g. "fix(controller): Updates such and such. Fixes #1234".
• I've signed my commits with DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My builds are green. Try syncing with master if they are not.
• My organization is added to USERS.md.

Currently, the order of the two steps for updating ephemeral metadata can cause some stable pods to end up having the old ephemeral metadata. As per the comment in the code, if we are fetching the set of pods first and updating them ----> then updating the replicaset template spec, then any pods created in the interim will be using the un-updated replicaset template spec, that is, the old ephemeral metadata.

And, since we have at the beginning of the function

modifiedRS, modified := replicasetutil.SyncReplicaSetEphemeralPodMetadata(rs, podMetadata)
if !modified {
  return nil
}

The next sync won't go in and update those dangling pods, since the replicaset is actually updated and hasn't been modified.

This happened with me on a scale of 500 pods, with Karpenter consolidation active evicting pods and causing new ones to be created, where this sort of race condition is likely to happen.

[github-actions]

Published E2E Test Results

  4 files    4 suites   3h 15m 11s ⏱️
113 tests 104 ✅  7 💤 2 ❌
454 runs  424 ✅ 28 💤 2 ❌

For more details on these failures, see this check.

Results for commit 4a0dc71.

♻️ This comment has been updated with latest results.

[github-actions]

Published Unit Test Results

2 280 tests   2 280 ✅  2m 59s ⏱️
  128 suites      0 💤
    1 files        0 ❌

Results for commit 4a0dc71.

♻️ This comment has been updated with latest results.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[codecov]

Codecov Report

Attention: Patch coverage is 73.33333% with 4 lines in your changes missing coverage. Please review.

Project coverage is 82.72%. Comparing base (5f59344) to head (4a0dc71).
Report is 13 commits behind head on master.

Files with missing lines	Patch %	Lines
rollout/ephemeralmetadata.go	73.33%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3975      +/-   ##
==========================================
+ Coverage   82.69%   82.72%   +0.02%     
==========================================
  Files         163      163              
  Lines       22895    22903       +8     
==========================================
+ Hits        18934    18947      +13     
+ Misses       3087     3084       -3     
+ Partials      874      872       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.