Fix HPA race condition by reading deployment replicas instead of HPA status

[ciarams87]

Proposed changes

Problem: When autoscaling.enable: true is configured in the Helm chart, the NGF controller updates the deployment and modifies the spec.replicas field in conflict with the HPA. This causes the deployment to scale up and down in the same second, resulting in constant pod churn and preventing the HPA from scaling up or down consistently.

Solution: When HPA is enabled, read the current Deployment.Spec.Replicas directly instead of HPA.Status.DesiredReplicas, which is eventually consistent and lags behind deployment changes. This prevents the controller from overwriting HPA's replica count with stale values, eliminating pod churn and connection drops.

Testing: Unit and local testing

Please focus on (optional): If you any specific areas where you would like reviewers to focus their attention or provide
specific feedback, add them here.

Closes #4007

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have checked that all unit tests pass after adding my changes
• I have updated necessary documentation
• I have rebased my branch onto main
• I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

Release notes

If this PR introduces a change that affects users and needs to be mentioned in the release notes,
please add a brief note that summarizes the change.

Preserve HPA replicas on deployment

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.12%. Comparing base (a5a0f72) to head (164a501).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4214      +/-   ##
==========================================
+ Coverage   86.10%   86.12%   +0.02%     
==========================================
  Files         131      131              
  Lines       14162    14171       +9     
  Branches       35       35              
==========================================
+ Hits        12194    12205      +11     
+ Misses       1765     1764       -1     
+ Partials      203      202       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[bjee19]

Have you verified that the situation described in the bug report was resolved?

[salonichf5]

Have you verified that the situation described in the bug report was resolved?

Yeah we should try to bottleneck the reconciliation/patch logic and see how that goes(stability eventually happens) since this was a bigger pain point for prod environments

[salonichf5]

🚀

[salonichf5]

Have you verified that the situation described in the bug report was resolved?

Yeah we should try to bottleneck the reconciliation/patch logic and see how that goes(stability eventually happens) since this was a bigger pain point for prod environments

The comment above answered my question.

[ciarams87]

Have you verified that the situation described in the bug report was resolved?

@bjee19 @salonichf5 To the best of my abilities, yes.

I created a HPA with min replicas of 1 and max of 5, and set replicas in the NginxProxy as 2. I manually changed the replicas to 5 using kubectl scale, verified that HPA scaled back to 1, and that the correct manager was managing the replica field (i.e. kube-controller-manager):

k get deployments.apps gateway-nginx --show-managed-fields -o json | \
  jq '.metadata.managedFields[] | select(.fieldsV1."f:spec"."f:replicas")'
{
  "apiVersion": "apps/v1",
  "fieldsType": "FieldsV1",
  "fieldsV1": {
    "f:spec": {
      "f:replicas": {}
    }
  },
  "manager": "kube-controller-manager",
  "operation": "Update",
  "subresource": "scale"
}