[Frontend] Support override generation config in args

[liuyanyi]

Support override generation config in args

In my past pr #11164, generation config could be load from model or other file.

In this pr, generation config can be override by user in args or model config, this allow control generation config in cli args.

override_generation_config has been added to ModelConfig and EngineArgs. A test has been added too.

Example Usage:

vllm serve Qwen/Qwen2.5-1.5B-Instruct --override-generation-config "{"top_k": 5}"

log will shows

INFO 01-24 23:05:28 serving_chat.py:90] Overwriting default chat sampling param with: {'top_k': 5}
INFO 01-24 23:05:28 serving_completion.py:54] Overwriting default completion sampling param with: {'top_k': 5}

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[DarkLight1337]

Can you explain the use of "auto" here?

[liuyanyi]

you mean the "auto" for "--generation-config"?
if auto is set, will load generation config from model dir, "--override-generation-config" allow user to manually set the generation config.
I think the newly added test should show the behavior between these parameters

[DarkLight1337]

Yes, can you update the help string to include this information?

[DarkLight1337]

Thanks for improving on this!

[DarkLight1337]

Please merge from main to fix the merge conflicts.

[liuyanyi]

Please merge from main to fix the merge conflicts.

Done

[ChewKokWah]

Support override generation config in args

In my past pr #11164, generation config could be load from model or other file.

In this pr, generation config can be override by user in args or model config, this allow control generation config in cli args.

override_generation_config has been added to ModelConfig and EngineArgs. A test has been added too.

Example Usage:

vllm serve Qwen/Qwen2.5-1.5B-Instruct --override-generation-config "{"top_k": 5}"

log will shows

INFO 01-24 23:05:28 serving_chat.py:90] Overwriting default chat sampling param with: {'top_k': 5}
INFO 01-24 23:05:28 serving_completion.py:54] Overwriting default completion sampling param with: {'top_k': 5}

1. When we had already set our own generation config in vllm.SamplingParams(), do we need to set again the sampling parameter with override-generation-config in order to override generation_config.json ? I used to thought that setting vllm.SamplingParams() will override generation_config.json, am I correct?

2. If we set 2 different parameter in vllm.SamplingParams() vs override-generation-config, which one will vLLM use?

[DarkLight1337]

--override-generation-config only sets the default sampling params. Per-request sampling params will override this.