[Feature] Skip modules for disabled modalities

[DarkLight1337]

🚀 The feature, motivation and pitch

Provide a way to skip loading modules by modality.

We can enable this automatically via --limit-mm-per-prompt, as discussed in #20868 (comment). For example, we can use text-only mode for an image-text model by setting --limit-mm-per-prompt '{"image": 0}'

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[sfeng33]

I can help on this
/assign