[RFC]: Refactor CI/CD

[robertgshaw2-redhat]

Motivation.

vLLM's CI/CD has grown in a less than ideal way as it has built up over the years.

We have the following problems:

• CI takes very long, especially on a per commit cycle
• CI has failures that cannot be reproduced on every machine due to numerics
• CI has failures on models that are not the 80-20 of our usage --- which runs per commit
• CI failures in early tests often lead to vLLM not cleaning up properly --- which creates failures across many tests that makes it hard to identify what is wrong
• CI is NOT covering the models that actually matter (since not enough GPU memory) or hardware that actually matters to the majority of our users
• CI is NOT covering performance!

These issues are creating a bad developer experience for vLLM and causes issues like the CI "death spiral", where we get into a cycle of force-merges.

These issues are creating challenges for vLLM's multiple HW backends, as we are unable to get reliable signal that keeps blocking issues.

Proposed Change.

The CI needs to be completely refactored and the culture around vLLM CI needs to be significantly improved.

Overall Goals:

• Remove V0 tests and migrate any missing coverage into V1
• Reduce per-commit CI to <15 minutes end-to-end
• Refactor all tests using golden strings or numerics to a stable status
• Refocus model testing onto the top 10 models that matter to 99% of users
• Acquire hardware resources that match the deployment patterns which matter to our users

This will require a concerted effort from the vLLM community.

We will appreciate anyone's help in executing on this effort. We are starting next week.

Some good ideas also in this issue: #20218

Feedback Period.

3 days

CC List.

@russellb @njhill @kushanam @shajrawi @simon-mo @andy-neuma @dougbtv

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[shajrawi]

cc @gshtras , @Alexei-V-Ivanov-AMD

[huachenheli]

cc @yeqcharlotte @houseroad

[kushanam]

Something we keep running into is making sure we actually test the same model with different parameters. Just running a static version doesn’t always catch bugs that pop up with certain server setups or parallel schemes and really testing every combo would get expensive fast. Maybe we could make a “proxy” version of each important model (with way fewer layers) and use those to run lots of aggressive tests across all the parameter variations. That way, we stress the same codepaths without breaking the bank.
cc: @cliffwoolley

[njhill]

Some immediately actionable ideas:

• Try to configure buildkite to retry individual failed tests and then abort the entire CI job rather than continuing to run other tests.
• Abort entire CI job if pre-commit fails (since it will need to be re-run regardless)
• Add timeouts to top level tests based on rough expected running time (plus some buffer)
• Replace use of models with smaller models (like opt-125m) for tests where the particular model isn't relevant
• Assuming we have locally-cached models, use HF_HUB_OFFLINE to avoid the (unreliable) remote connection to hf hub
• Host images used in multi-model tests locally. Example failure.

Other suggestions:

• Look for big wins tied to longest-running tests
• Audit / reorganize tests for more systematic coverage
• Look for tests that can be grouped to reduce the number of server starts/stops (where we launch vllm with the same config or where the same config could be used without affecting the nature of the test)