[Misc] Reduce initialization time of auto_tune

[wdhongtw]

Each iteration of OOM execution when finding max available memory-utilization results in 10 min delay.
Try to detect crashed situation and continue to next iteration efficiently.

Purpose

Currently when auto_tune.sh is scanning max available value of memory-utilization,
it takes unnecessary time and it's kinda confusing. Since that user usually sees no
child process in that bash session for about 10 * N min. N is the number of iterations
until finding the available value.

This PR help reducing this confusing situation and enhance the efficiency by proactively
detects the OOM situation and continue to next iteration.

Test Plan

No test plan for this change, as this PR is related to helper script.

Test Result

N/A

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

[gemini-code-assist]

Code Review

This pull request improves the auto_tune.sh script by detecting server crashes early, which reduces unnecessary delays during the auto-tuning process. The approach of checking the server's process ID is sound. However, it could be made more robust to prevent potential issues with PID reuse, which could lead to incorrect benchmark results. I've suggested a small change to also verify the process command line, making the check more reliable.

[Chenyaaang]

LGTM, thanks for the improvement!

[yaochengji]

LGTM, thanks for the improvement!