[Core][Doc] Default to multiprocessing for single-node distributed case

[njhill]

Also update docs to reflect support for the multiprocessing distributed executor.

Resolves #4955
Resolves #5221

[youkaichao]

LGTM, thanks!

[Yard1]

If we detect we are in a Ray placement group (ray.util.get_current_placement_group() is not None), we should still use Ray by default.

[rcarrata]

LGTM, thanks @njhill !!

[vrdn-23]

Would it possible to also make ray optional or as an extra when installing vllm as part of this PR? Currently it's being installed by default as part of the requirements for vllm so it'll always be present?

[njhill]

@vrdn-23 I had considered this and actually have a change for it in a branch since this is what we do in our own builds. But right now there are other dependencies included that could be considered optional as well, such as pillow, outlines, lm_format_enforcer so we should perhaps make those optional extras too. The only one managed that way right now is tensorizer.

Would be good to get thoughts from others on this too.

[rcarrata]

with all these dependencies (not only affect ray, also other components that may cause any issue), I'd suggest to open another PR from the branch that is suggested by @njhill alongside another GH issue to track the effort.

[simon-mo]

+1 I'm in favor of optional dependencies and trimming down our dependencies in general. Please open PRs!

[simon-mo]

I think these are actual failures

https://buildkite.com/vllm/ci-aws/builds/646#018ff31c-991c-439e-b417-efee7c59e5c0

[robertgshaw2-neuralmagic]

+1