[serve] Fix bug with 'proxy_location' set for 'serve run' CLI command + discrepancy fix in Python API 'serve.start' function #57622

axreldable · 2025-10-10T09:18:41Z

Why are these changes needed?

Fix bug with 'proxy_location' set for 'serve run' CLI command

serve run CLI command ignores proxy_location from config and uses default value EveryNode.

Steps to reproduce:

have a script:

# hello_world.py
from ray.serve import deployment

@deployment
async def hello_world():
    return "Hello, world!"

hello_world_app = hello_world.bind()

Execute:

ray stop
ray start --head
serve build -o config.yaml hello_world:hello_world_app

change proxy_location in the config.yaml: EveryNode -> Disabled

serve run config.yaml
curl -s -X GET "http://localhost:8265/api/serve/applications/" | jq -r '.proxy_location'

Output:

Before change:
EveryNode - but Disabled expected
After change:
Disabled

Fix discrepancy for 'proxy_location' in the Python API 'start' method

serve.start function in Python API sets different http_options.location depending on if http_options is provided.

Steps to reproduce:

have a script:

# discrepancy.py
import time

from ray import serve
from ray.serve.context import _get_global_client

if __name__ == '__main__':
    serve.start()
    client = _get_global_client()
    print(f"Empty http_options: `{client.http_config.location}`")

    serve.shutdown()
    time.sleep(5)

    serve.start(http_options={"host": "0.0.0.0"})
    client = _get_global_client()
    print(f"Non empty http_options: `{client.http_config.location}`")

Execute:

ray stop
ray start --head
python -m discrepancy

Output:

Before change:
Empty http_options: `EveryNode`
Non empty http_options: `HeadOnly`
After change:
Empty http_options: `EveryNode`
Non empty http_options: `EveryNode`

It changes current behavior in the following ways:

serve run CLI command respects proxy_location parameter from config instead of using the hardcoded EveryNode.
serve.start function in Python API stops using the default HeadOnly in case of empty proxy_location and provided http_options dictionary without location specified.

Related issue number

Aims to simplify changes in the PR: #56507

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run pre-commit jobs to lint the changes in this PR. (pre-commit setup)
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: axreldable <aleksei.starikov.ax@gmail.com>

…' method Signed-off-by: axreldable <aleksei.starikov.ax@gmail.com>

Signed-off-by: axreldable <aleksei.starikov.ax@gmail.com>

axreldable · 2025-10-10T17:01:35Z

python/ray/serve/tests/conftest.py

        _system_config={"metrics_report_interval_ms": 1000, "task_retry_delay_ms": 50},
    )
    serve.start(
+        proxy_location=ProxyLocation.HeadOnly,


This is the current value which is used implicitly.

just confirming, even if we remove this, serve.start will still use HeadOnly?

It was true before the change. In the current master, the result proxy_location if not provided explicitly depends on the presence of http_options in serve.start parameters. Empty http_options gives EveryNode, but non-empty http_options gives HeadOnly.

This PR changes this discrepancy. After the change serve.start runs cluster with default EveryNode proxy_location in case of empty proxy_location parameter.

See the example in the description:

Fix discrepancy for 'proxy_location' in the Python API 'start' method

serve.start function in Python API sets different http_options.location depending on if http_options is provided.

Steps to reproduce:

have a script:

# discrepancy.py import time from ray import serve from ray.serve.context import _get_global_client if __name__ == '__main__': serve.start() client = _get_global_client() print(f"Empty http_options: `{client.http_config.location}`") serve.shutdown() time.sleep(5) serve.start(http_options={"host": "0.0.0.0"}) client = _get_global_client() print(f"Non empty http_options: `{client.http_config.location}`")

Execute:

ray stop ray start --head python -m discrepancy

Output:

Before change: Empty http_options: `EveryNode` Non empty http_options: `HeadOnly` After change: Empty http_options: `EveryNode` Non empty http_options: `EveryNode`

So, I use now

serve.start( proxy_location=ProxyLocation.HeadOnly,

to have the same behavior in tests as before the change.

makes sense :)

Signed-off-by: axreldable <aleksei.starikov.ax@gmail.com>

harshit-anyscale

lgtm

axreldable · 2025-10-20T08:43:11Z

Thank you for the approval, @harshit-anyscale !
@abrarsheikh , could you please review this as well?

abrarsheikh

does serve run restarts the cluster? I am trying to understand what happens if we call serve run the very first time with location = HeadOnly, then call serve run again with location = EveryNode. How does this change percolate through the cluster.

Is the entire cluster restarted include controller, proxies? If not should be disallow changing location during the second serve run command?

abrarsheikh · 2025-10-26T05:55:17Z

python/ray/serve/_private/config.py

+def prepare_http_options(
+    proxy_location: Union[None, str, ProxyLocation],
+    http_options: Union[None, dict, HTTPOptions],
+) -> HTTPOptions:


We are using this function to consolidate http_options, using http_options and proxy_location passed by user. Which is well intended, but the issue is that

In imperative mode, user passes HTTPOptions

In declarative mode user passes HTTPOptionsSchema

I think we should keep this function specific to imperative mode.

Ok, I can remove the usage of the prepare_http_options func from scripts.py and use it in serve.start api only. Will it work?

And could you please clarify what do you mean by imperative and declarative modes?

imperative ->serve.run(app)
declarative -> serve deploy config.yaml

Thank you for the clarification! I renamed prepare_http_options -> prepare_imperative_http_options and left a note about the distinction in the doc string.

abrarsheikh · 2025-10-26T06:01:11Z

python/ray/serve/scripts.py

+    proxy_location = None
+    http_options = None
    grpc_options = gRPCOptions()
    # Merge http_options and grpc_options with the ones on ServeDeploySchema.
    if is_config and isinstance(config, ServeDeploySchema):
-        config_http_options = config.http_options.dict()
-        http_options = {**config_http_options, **http_options}
+        proxy_location = config.proxy_location
+        http_options = config.http_options.dict()


if the core issue we are trying to solve is that serve run does not pick proxy_location from config, then is the following sufficient to fix that?

http_options = {"location": "EveryNode"} grpc_options = gRPCOptions() # Merge http_options and grpc_options with the ones on ServeDeploySchema. if is_config and isinstance(config, ServeDeploySchema): http_options["location"] = config.proxy_location config_http_options = config.http_options.dict() http_options = {**config_http_options, **http_options} grpc_options = gRPCOptions(**config.grpc_options.dict()) client = _private_api.serve_start( http_options=http_options, .....

Almost, this will be sufficient:

http_options = {"location": "EveryNode"} grpc_options = gRPCOptions() # Merge http_options and grpc_options with the ones on ServeDeploySchema. if is_config and isinstance(config, ServeDeploySchema): http_options["location"] = ProxyLocation._to_deployment_mode(config.proxy_location).value config_http_options = config.http_options.dict() http_options = {**config_http_options, **http_options} grpc_options = gRPCOptions(**config.grpc_options.dict())

http_options["location"] = config.proxy_location ---> http_options["location"] = ProxyLocation._to_deployment_mode(config.proxy_location).value

makes sense, then let's make this change

Applied the change.

abrarsheikh · 2025-10-26T06:04:33Z

python/ray/serve/tests/conftest.py

        _system_config={"metrics_report_interval_ms": 1000, "task_retry_delay_ms": 50},
    )
    serve.start(
+        proxy_location=ProxyLocation.HeadOnly,


just confirming, even if we remove this, serve.start will still use HeadOnly?

axreldable · 2025-10-26T10:59:02Z

does serve run restarts the cluster? I am trying to understand what happens if we call serve run the very first time with location = HeadOnly, then call serve run again with location = EveryNode. How does this change percolate through the cluster.

Is the entire cluster restarted include controller, proxies? If not should be disallow changing location during the second serve run command?

does serve run restarts the cluster?

No, the run command goes to serve._private.serve_start which starts cluster or returns existing ServeControllerClient.

I am trying to understand what happens if we call serve run the very first time with location = HeadOnly, then call serve run again with location = EveryNode. How does this change percolate through the cluster.

Cluster will not be restarted and continue running with location = HeadOnly. Info and warning messages will be logged about the attempt to change http options for the cluster.

However, you can't right now start cluster with the cli run command with location = HeadOnly configured in config file. It will be overridden with EveryNode. This PR fixes it.

Is the entire cluster restarted include controller, proxies? If not should be disallow changing location during the second serve run command?

Correct, I'm working on exactly that in this PR - Fail on the change of 'proxy_location' or 'http_options' parameters for the 'serve' API. To simplify the change, I opened this PR to resolve the bug first. As the logic became too complicated there with this bug and discrepancy.

…name to prepare_imperative_http_options Signed-off-by: axreldable <aleksei.starikov.ax@gmail.com>

…_http_options_5

axreldable · 2025-10-29T12:57:11Z

Hi @abrarsheikh ! I addressed comments. Could you please check?

abrarsheikh · 2025-10-29T17:10:48Z

python/ray/serve/tests/conftest.py

        _system_config={"metrics_report_interval_ms": 1000, "task_retry_delay_ms": 50},
    )
    serve.start(
+        proxy_location=ProxyLocation.HeadOnly,


makes sense :)

@deployment

… + discrepancy fix in Python API 'serve.start' function (ray-project#57622)   ## Why are these changes needed? 1. Fix bug with 'proxy_location' set for 'serve run' CLI command `serve run` CLI command ignores `proxy_location` from config and uses default value `EveryNode`. Steps to reproduce: - have a script: ```python # hello_world.py from ray.serve import deployment @deployment async def hello_world(): return "Hello, world!" hello_world_app = hello_world.bind() ``` Execute: ``` ray stop ray start --head serve build -o config.yaml hello_world:hello_world_app ``` - change `proxy_location` in the `config.yaml`: EveryNode -> Disabled ``` serve run config.yaml curl -s -X GET "http://localhost:8265/api/serve/applications/" | jq -r '.proxy_location' ``` Output: ``` Before change: EveryNode - but Disabled expected After change: Disabled ``` 2. Fix discrepancy for 'proxy_location' in the Python API 'start' method `serve.start` function in Python API sets different `http_options.location` depending on if `http_options` is provided. Steps to reproduce: - have a script: ```python # discrepancy.py import time from ray import serve from ray.serve.context import _get_global_client if __name__ == '__main__': serve.start() client = _get_global_client() print(f"Empty http_options: `{client.http_config.location}`") serve.shutdown() time.sleep(5) serve.start(http_options={"host": "0.0.0.0"}) client = _get_global_client() print(f"Non empty http_options: `{client.http_config.location}`") ``` Execute: ``` ray stop ray start --head python -m discrepancy ``` Output: ``` Before change: Empty http_options: `EveryNode` Non empty http_options: `HeadOnly` After change: Empty http_options: `EveryNode` Non empty http_options: `EveryNode` ``` ------------------------------------------------------------- It changes current behavior in the following ways: 1. `serve run` CLI command respects `proxy_location` parameter from config instead of using the hardcoded `EveryNode`. 2. `serve.start` function in Python API stops using the default `HeadOnly` in case of empty `proxy_location` and provided `http_options` dictionary without `location` specified.  ## Related issue number  Aims to simplify changes in the PR: ray-project#56507 ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run pre-commit jobs to lint the changes in this PR. ([pre-commit setup](https://docs.ray.io/en/latest/ray-contribute/getting-involved.html#lint-and-formatting)) - [x] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [x] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [x] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: axreldable <aleksei.starikov.ax@gmail.com>

@deployment

… + discrepancy fix in Python API 'serve.start' function (ray-project#57622)   ## Why are these changes needed? 1. Fix bug with 'proxy_location' set for 'serve run' CLI command `serve run` CLI command ignores `proxy_location` from config and uses default value `EveryNode`. Steps to reproduce: - have a script: ```python # hello_world.py from ray.serve import deployment @deployment async def hello_world(): return "Hello, world!" hello_world_app = hello_world.bind() ``` Execute: ``` ray stop ray start --head serve build -o config.yaml hello_world:hello_world_app ``` - change `proxy_location` in the `config.yaml`: EveryNode -> Disabled ``` serve run config.yaml curl -s -X GET "http://localhost:8265/api/serve/applications/" | jq -r '.proxy_location' ``` Output: ``` Before change: EveryNode - but Disabled expected After change: Disabled ``` 2. Fix discrepancy for 'proxy_location' in the Python API 'start' method `serve.start` function in Python API sets different `http_options.location` depending on if `http_options` is provided. Steps to reproduce: - have a script: ```python # discrepancy.py import time from ray import serve from ray.serve.context import _get_global_client if __name__ == '__main__': serve.start() client = _get_global_client() print(f"Empty http_options: `{client.http_config.location}`") serve.shutdown() time.sleep(5) serve.start(http_options={"host": "0.0.0.0"}) client = _get_global_client() print(f"Non empty http_options: `{client.http_config.location}`") ``` Execute: ``` ray stop ray start --head python -m discrepancy ``` Output: ``` Before change: Empty http_options: `EveryNode` Non empty http_options: `HeadOnly` After change: Empty http_options: `EveryNode` Non empty http_options: `EveryNode` ``` ------------------------------------------------------------- It changes current behavior in the following ways: 1. `serve run` CLI command respects `proxy_location` parameter from config instead of using the hardcoded `EveryNode`. 2. `serve.start` function in Python API stops using the default `HeadOnly` in case of empty `proxy_location` and provided `http_options` dictionary without `location` specified.  ## Related issue number  Aims to simplify changes in the PR: ray-project#56507 ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run pre-commit jobs to lint the changes in this PR. ([pre-commit setup](https://docs.ray.io/en/latest/ray-contribute/getting-involved.html#lint-and-formatting)) - [x] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [x] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [x] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: axreldable <aleksei.starikov.ax@gmail.com>

@deployment

… + discrepancy fix in Python API 'serve.start' function (ray-project#57622)   ## Why are these changes needed? 1. Fix bug with 'proxy_location' set for 'serve run' CLI command `serve run` CLI command ignores `proxy_location` from config and uses default value `EveryNode`. Steps to reproduce: - have a script: ```python # hello_world.py from ray.serve import deployment @deployment async def hello_world(): return "Hello, world!" hello_world_app = hello_world.bind() ``` Execute: ``` ray stop ray start --head serve build -o config.yaml hello_world:hello_world_app ``` - change `proxy_location` in the `config.yaml`: EveryNode -> Disabled ``` serve run config.yaml curl -s -X GET "http://localhost:8265/api/serve/applications/" | jq -r '.proxy_location' ``` Output: ``` Before change: EveryNode - but Disabled expected After change: Disabled ``` 2. Fix discrepancy for 'proxy_location' in the Python API 'start' method `serve.start` function in Python API sets different `http_options.location` depending on if `http_options` is provided. Steps to reproduce: - have a script: ```python # discrepancy.py import time from ray import serve from ray.serve.context import _get_global_client if __name__ == '__main__': serve.start() client = _get_global_client() print(f"Empty http_options: `{client.http_config.location}`") serve.shutdown() time.sleep(5) serve.start(http_options={"host": "0.0.0.0"}) client = _get_global_client() print(f"Non empty http_options: `{client.http_config.location}`") ``` Execute: ``` ray stop ray start --head python -m discrepancy ``` Output: ``` Before change: Empty http_options: `EveryNode` Non empty http_options: `HeadOnly` After change: Empty http_options: `EveryNode` Non empty http_options: `EveryNode` ``` ------------------------------------------------------------- It changes current behavior in the following ways: 1. `serve run` CLI command respects `proxy_location` parameter from config instead of using the hardcoded `EveryNode`. 2. `serve.start` function in Python API stops using the default `HeadOnly` in case of empty `proxy_location` and provided `http_options` dictionary without `location` specified.  ## Related issue number  Aims to simplify changes in the PR: ray-project#56507 ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run pre-commit jobs to lint the changes in this PR. ([pre-commit setup](https://docs.ray.io/en/latest/ray-contribute/getting-involved.html#lint-and-formatting)) - [x] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [x] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [x] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: axreldable <aleksei.starikov.ax@gmail.com> Signed-off-by: Aydin Abiar <aydin@anyscale.com>

axreldable added 5 commits October 6, 2025 22:47

[serve] Dummy change

00eb339

Signed-off-by: axreldable <aleksei.starikov.ax@gmail.com>

Merge remote-tracking branch 'origin/master'

f6e8303

[serve] Dummy change

5192d56

Signed-off-by: axreldable <aleksei.starikov.ax@gmail.com>

Merge remote-tracking branch 'origin/master'

47c563b

[serve] Fix discrepancy for 'proxy_location' in the Python API 'start…

d978d26

…' method Signed-off-by: axreldable <aleksei.starikov.ax@gmail.com>

axreldable force-pushed the 56163_http_options_5 branch 2 times, most recently from 3de646d to 3cdf062 Compare October 10, 2025 13:13

[serve] Fix bug with 'proxy_location' set for 'serve run' CLI command

b4c6107

Signed-off-by: axreldable <aleksei.starikov.ax@gmail.com>

axreldable force-pushed the 56163_http_options_5 branch from 3cdf062 to b4c6107 Compare October 10, 2025 16:02

axreldable commented Oct 10, 2025

View reviewed changes

axreldable changed the title ~~56163 http options 5~~ [serve] Fix bug with 'proxy_location' set for 'serve run' CLI command + discrepancy fix in Python API 'serve.start' function Oct 10, 2025

axreldable marked this pull request as ready for review October 10, 2025 19:28

axreldable requested a review from a team as a code owner October 10, 2025 19:28

axreldable mentioned this pull request Oct 10, 2025

[serve] Fix bug with 'proxy_location' set for 'serve run' CLI command + discrepancy fix in Python API 'serve.start' function #56993

Closed

8 tasks

This comment was marked as outdated.

Sign in to view

axreldable mentioned this pull request Oct 10, 2025

[serve] Fail on the change of 'proxy_location' or 'http_options' parameters for the 'serve' API #56507

Open

8 tasks

ray-gardener bot added serve Ray Serve Related Issue community-contribution Contributed by the community labels Oct 11, 2025

[serve] Refactor prepare_http_options method

3d44bf7

Signed-off-by: axreldable <aleksei.starikov.ax@gmail.com>

axreldable force-pushed the 56163_http_options_5 branch from b63089c to 3d44bf7 Compare October 11, 2025 15:10

This comment was marked as outdated.

Sign in to view

[serve] Add tests for prepare_http_options method

0b59f32

Signed-off-by: axreldable <aleksei.starikov.ax@gmail.com>

This comment was marked as outdated.

Sign in to view

axreldable added 2 commits October 11, 2025 21:06

[serve] Fix test_proxy_location test

72f7b60

Signed-off-by: axreldable <aleksei.starikov.ax@gmail.com>

Merge branch 'master' into 56163_http_options_5

8f9efd9

harshit-anyscale added the go add ONLY when ready to merge, run all tests label Oct 13, 2025

harshit-anyscale approved these changes Oct 13, 2025

View reviewed changes

Merge branch 'master' into 56163_http_options_5

0040b9c

abrarsheikh reviewed Oct 26, 2025

View reviewed changes

axreldable added 3 commits October 29, 2025 12:29

[serve] Use prepare_http_options method only for imperative mode + re…

d0ff73c

…name to prepare_imperative_http_options Signed-off-by: axreldable <aleksei.starikov.ax@gmail.com>

Merge remote-tracking branch 'origin/56163_http_options_5' into 56163…

96b63be

…_http_options_5

Merge branch 'master' into 56163_http_options_5

969f07e

axreldable requested a review from abrarsheikh October 29, 2025 12:57

abrarsheikh approved these changes Oct 29, 2025

View reviewed changes

abrarsheikh merged commit 92d8471 into ray-project:master Oct 29, 2025
6 checks passed

axreldable deleted the 56163_http_options_5 branch October 29, 2025 18:38

[serve] Fix bug with 'proxy_location' set for 'serve run' CLI command + discrepancy fix in Python API 'serve.start' function #57622

[serve] Fix bug with 'proxy_location' set for 'serve run' CLI command + discrepancy fix in Python API 'serve.start' function #57622

Uh oh!

Conversation

axreldable commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

axreldable Oct 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

harshit-anyscale left a comment

Choose a reason for hiding this comment

Uh oh!

axreldable commented Oct 20, 2025

Uh oh!

abrarsheikh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

axreldable commented Oct 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

axreldable commented Oct 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

axreldable commented Oct 10, 2025 •

edited

Loading

axreldable Oct 26, 2025 •

edited

Loading

axreldable commented Oct 26, 2025 •

edited

Loading