[Perf] Use small max_num_batched_tokens for A100

[KuntaiDu]

This PR solves a performance regression issue on A100. Related PR: #17073

	Token throughput (Before this PR)	Token throughput (After this PR)
Llama 70B, TP=2	1054.01	1438.49 (+36%)
Llama 70B, TP=4	3053.03	2994.03 (-2%)

vLLM launching command:
TP=2:

vllm serve meta-llama/Llama-3.1-70B-Instruct     --port 8000     -tp 2     --max-model-len 10000     --gpu-memory-utilization 0.95     --swap-space 0

TP=4:

vllm serve meta-llama/Llama-3.1-70B-Instruct     --port 8000     -tp 4     --max-model-len 10000     --gpu-memory-utilization 0.95     --swap-space 0

Benchmarking command:

python benchmark_serving.py --port 8000     --dataset-name sharegpt     --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json     --num-prompts 400     --request-rate inf     --model meta-llama/Llama-3.1-70B-Instruct

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[hmellor]

It might make sense to move all these heuristics into the respective platform file?

As it is right now we have side effects where non-GPU systems (except TPU) are having their defaults set based on memory heuristics meant for GPUs. i.e. if you have a CPU platform with >70GB RAM you're get the large GPU defaults for max_num_batched_tokens...

edit: I've just noticed this is V1 only (no CPU support yet), but this will probably cause weird behaviour once we do support CPU in V1

[DarkLight1337]

Can you merge from main to fix pre-commit?

[heheda12345]

Would cudaDeviceProp.multiProcessorCount be a better metric? We can get this metric from numba.

[KuntaiDu]

Let me fix the precommit

[KuntaiDu]

The CI seems to stall, rebuild the CI to try to make it green.

[MingzhenHan]

@KuntaiDu Hello Kuntai, Why do we need to use smaller max_num_batched_tokens on A100? What is the reason for this? Looking forward to your guidance🙏