[BUG]: Health Status Transition Issue: Services Stuck in "notready" State

[GavinZhu-GMI]

Describe the Bug

Problem Description

Current Issue: Dynamo services using Global Health Mode never automatically transition from "notready" to "ready" status, causing production deployment failures and requiring manual workarounds.

---

Current Workaround

# Required workaround in every deployment
env:
  - name: DYN_SYSTEM_STARTING_HEALTH_STATUS
    value: "ready"  # Forces immediate ready state

Steps to Reproduce

1. Deploy a Dynamo service without explicit endpoint health configuration
2. Check health endpoint: curl http://service:9090/health

Expected Behavior

Expected: Service transitions to "ready" after successful initialization

Actual Behavior

Actual: Service permanently returns {"status":"notready"} with HTTP 503

Environment

Two node PD disagg H200.

Additional Context

No response

Screenshots

No response

[GavinZhu-GMI]

The only manual method is set_health_status(), which is not called.
Hence I think a transition logic is in need, I did some initial coding and I need to learn more about the design purpose of DYN_SYSTEM_STARTING_HEALTH_STATUS to move forward.

[GavinZhu-GMI]

@nnshah1 I would really appreciate your input here.

[nnshah1]

@GavinZhu-GMI - we are currently in process of updating the health check to be more explicit by adding an api to expose a canary check tied to service endpoint being available.

If needed we can also enable a public api to set the status - in addition to the default - are there conditions outside of an endpoint that need to be checked for health?

Can you check if this would address your needs:

#2903

@tzulingk for visability.

[nnshah1]

@GavinZhu-GMI can you check the latest release to see if it meets your needs? the health checks have been updated - if there is still a need for a generic ability for a service to set the health status we can look into adding that

[GavinZhu-GMI]

It's fixed, thanks!