[serve] mark proxy as unready when its routers are aware of zero replicas

[zcin]

Why are these changes needed?

Pinging /-/routes and /-/healthz returns 503 if proxy is unavailable, signaling that the proxy is not ready to serve traffic.
Previously the proxy would return 503 if the route table has not been populated yet. This PR makes it so that proxy will also return a 503 if it hasn't received any replicas.

Specifically (wrt routing):

• If route table is not populated, proxy is not ready / unavailable.
• If (proxy is on a worker node, and) route table is populated, but none of the handles (corresponding to each endpoint) have received a nonzero set of running replicas from controller, proxy is not ready.
  • Note that if proxy is on head node, proxy will be considered ready for traffic even if it doesn't have any replicas to handle the scale-to-zero case. If all deployments are scaled to zero, the head node proxy will be the only remaining proxy, and it will need to receive requests to trigger upscale.
• Otherwise, proxy is ready.

Related issue number

closes #46938

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[zcin]

moved to test_cluster.py

[zcin]

this is from test_standalone_3, unchanged

[edoakes]

Stylistic comments inline.

This needs to handle the scale-to-zero case, which I'm not seeing (let me know if missed). If all applications have min_replicas=0 (or there is only one), we could end up in a scenario where all proxies fail health checks and receive no traffic.

[edoakes]

should this also require that the size of the route_table is > 0?

[zcin]

yeah that makes sense, changed so that it is only flipped on if a nonzero list of routes were received.

[edoakes]

this name is misleading -- it could be that the running replicas have been populated at least once, but the size of the list is 0

suggest to call it something like has_running_replicas

[zcin]

hmm so right now what I've implemented is that the proxy is considered ready if it's received a non-zero list of replicas at least once, but if after that the number of replicas scale back down to 0, the flag isn't flipped off. For this I think running_replicas_populated is probably still more accurate. Implementing it this way targets more of the "proxy just came up and hasn't had a chance to receive replicas yet before head node goes down" case.

I can also change it so that if the number of replicas scale back down to 0, the flag is flipped off -> the proxy changes to unhealthy so that it won't receive traffic. This will cover more bases, but will put more pressure on the head node proxy when all ingress deployments have scaled to zero. WDYT?

[edoakes]

Got it. I think the existing naming (and behavior) is reasonable then.

[zcin]

This needs to handle the scale-to-zero case, which I'm not seeing (let me know if missed). If all applications have min_replicas=0 (or there is only one), we could end up in a scenario where all proxies fail health checks and receive no traffic.

Synced offline, the special case with the head node is meant to handle the scale-to-zero case. The head node proxy will not fail health check if its handles have zero replicas.