[Misc][Doc] Add note regarding loading generation_config by default

[ywang96]

#12622 introduced a change to apply generation_config, therefore can sometimes silently override default sampling parameters and cause user confusion. See #15241 as an example.

This PR updates the documentation regarding this change and gives a warning if any of the sampling parameters is overridden by the generation config.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[ywang96]

Server log from launching vllm serve Qwen/Qwen2.5-VL-7B-Instruct on this branch. I think this should be enough notice for the end user.

INFO 03-21 08:44:05 [core.py:138] init engine (profile, create kv cache, warmup model) took 45.93 seconds
WARNING 03-21 08:44:05 [config.py:1028] Default sampling parameters have been overridden by the model's huggingface generation config. Please pass `--generation-config vllm` at server launch if this is not intended.
INFO 03-21 08:44:05 [serving_chat.py:115] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.1, 'top_k': 1, 'top_p': 0.001}
INFO 03-21 08:44:05 [serving_completion.py:61] Using default completion sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.1, 'top_k': 1, 'top_p': 0.001}
INFO 03-21 08:44:05 [api_server.py:1028] Starting vLLM API server on http://0.0.0.0:8000

[hmellor]

I'm not sure that this is what's happening, let's continue the discussion on #15241

[DarkLight1337]

This LGTM but let's have @hmellor discuss with you first

[hmellor]

Ok I have figured out what wasn't right so we can bring the discussion back here.

#12622 changed the default behaviour of the config, so all entryoints had their default behaviour changed (not just online).

The behaviour observed in #15241 was because passing sampling parameters is handled differently in online and offline:

• online - the server's default generation config updates individual parameters with ones passed by the user
• offline - the LLM's default generation config is replaced with the SamplingParams passed by the user

  vllm/vllm/entrypoints/llm.py

  Lines 453 to 455 in 47c7126

  	if sampling_params is None:
  	# Use default sampling params.
  	sampling_params = self.get_default_sampling_params()

This means that when you passed SamplingParams(temperature=1) to LLM.generate it was silently removing all the other default sampling params. This difference in behaviour is what caused the confusion.

[hmellor]

This warning combined with the info log from the server makes it abundently clear to the user where the generation config is coming from and what the values are.

Could we:

• Verify that this warning appears when using the LLM class
• Add a warning to LLM.generate when sampling_parameters is passed explaining that this overrides all the default sampling params in LLM not just the ones specified in the passed argument

[ywang96]

Confirmed the warning appears when using LLM class.

For the warning, I think this is a bit tricky because if sampling_params is created from the way that we suggest in the example below, then it's not overriding all defaults

vllm/examples/offline_inference/basic/generate.py

Lines 18 to 26 in f68cce8

	sampling_params = llm.get_default_sampling_params()
	if max_tokens is not None:
	sampling_params.max_tokens = max_tokens
	if temperature is not None:
	sampling_params.temperature = temperature
	if top_p is not None:
	sampling_params.top_p = top_p
	if top_k is not None:
	sampling_params.top_k = top_k

.

I think on this note, we should probably make llm.get_default_sampling_params() as the post_init of SamplingParams, but we can leave that to a separate discussion.

[ywang96]

if {class}~vllm.SamplingParams is not specified.

SamplingParams is called out in particular here to differentiate this from online serving

[hmellor]

LGTM!

One small nit would be to change huggingface -> Hugging Face, also it looks like pre-commit is upset about some whitespace.