for glm-4.1V update

[zRzRzRzRzRzRzR]

This PR aims to upgrade the implementation of GLM-4.1V to be compatible with GLM-V implementations that use different base models but share the same VIT.
The current PR is incorrect, as discussed on Slack, because the transformers library architectures do not appear in the text config.

[gemini-code-assist]

Code Review

This pull request aims to make the GLM-4.1V implementation more flexible by supporting different base language models. However, the current changes hardcode a specific language model architecture, which contradicts the stated goal. Additionally, there's an incorrect entry in the model registry that could cause issues. My review focuses on these critical issues and provides suggestions to address them, ensuring the implementation is robust and achieves its intended purpose.

[gemini-code-assist]

This change incorrectly registers a multimodal architecture Glm4v_moeForConditionalGeneration as a text-generation model. This architecture is already correctly registered as a multimodal model in _MULTIMODAL_MODELS. This entry is likely a mistake and could cause conflicts or unexpected behavior. The module glm4v_moe_text also does not appear to exist in the codebase. This line should be removed.

[gemini-code-assist]

This block introduces two main issues:

1. Incorrect hf_config type: The hf_config argument is set to config.text_config. If config is a transformers.Glm4vConfig object, config.text_config is a dictionary, but init_vllm_registered_model expects a transformers.PretrainedConfig object. This will likely cause a TypeError. The previous implementation correctly used config.get_text_config(), which returns a PretrainedConfig object.

2. Hardcoded Architecture: The architectures parameter is hardcoded to ["Glm4MoeForCausalLM"]. This contradicts the PR's goal of supporting different base models. The architecture should be determined dynamically from the model's configuration, for example from config.text_config.architectures.

To address these issues, you should ensure hf_config is a PretrainedConfig object and avoid hardcoding the architecture. This will make the implementation more robust and aligned with the PR's objective.

        text_config = (config.get_text_config()
                       if hasattr(config, "get_text_config") else
                       config.text_config)

        # The `text_config` from a multi-modal model's config is often a
        # dictionary. We need to convert it to a PretrainedConfig object.
        if isinstance(text_config, dict):
            from transformers import AutoConfig
            text_config = AutoConfig.from_dict(text_config,
                                               trust_remote_code=True)

        self.language_model = init_vllm_registered_model(
            vllm_config=vllm_config,
            hf_config=text_config,
            prefix=maybe_prefix(prefix, "language_model"))

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[Isotr0py]

Some initial comments. Looking into the text backbone initialization issue.

[Isotr0py]

BTW, you can enable automatic sign-off in pycharm: https://docs.vllm.ai/en/latest/contributing/index.html?h=pycharm#dco-and-signed-off-by

[Isotr0py]

Nearly forgotten, can you also update tests/models/registry.py and docs/models/supported_models.md for Glm4v_moeForConditionalGeneration?

[DarkLight1337]

Please also merge from main to fix the CI

[Isotr0py]

Seems the failing multimodal tests are related to the registry renaming: https://buildkite.com/vllm/ci/builds/25800#019868cb-5bbe-41fc-be3b-f86a7d8edfc9

[zRzRzRzRzRzRzR]

Have the GLM model related tests have now passed ?

[DarkLight1337]

Yes, merging