Skip to content

Add VLM-aware model loading to FSDP→HF conversion#19

Open
cklxx wants to merge 6 commits intomainfrom
add-support-for-vlm-checkpoints-conversion
Open

Add VLM-aware model loading to FSDP→HF conversion#19
cklxx wants to merge 6 commits intomainfrom
add-support-for-vlm-checkpoints-conversion

Conversation

@cklxx
Copy link
Owner

@cklxx cklxx commented Jan 20, 2026

Motivation

  • Enable conversion of Vision-Language-Model (VLM) FSDP checkpoints to Hugging Face format by selecting the correct HF model class based on the model config.

Description

  • Updated tools/convert_fsdp_to_hf.py to import AutoModelForImageTextToText and added _build_hf_model(config) which prints the detected config.model_type and returns either AutoModelForCausalLM or AutoModelForImageTextToText using trust_remote_code=True.

Testing

  • Ran linting and formatting checks: ruff check ., black --check ., and isort --check ., all passed.
  • Ran pytest, which failed during collection with ModuleNotFoundError: No module named 'slime' (test environment import issue).

Codex Task

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5bcb474c90

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@cklxx cklxx force-pushed the add-support-for-vlm-checkpoints-conversion branch from 499c814 to 7627680 Compare January 21, 2026 04:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants