Is finetuning of LLaVA-CoT not available yet? (2024/11/26)

[Bleking]

Thank you for releasing this model. I tried finetuning this model with my custom dataset using this script from LLaVA. However, I think I will have to set the version, visual encoder, and so on in order to use the model for finetuning, so I don't think I can do that now.

So far, since it says "Stay tuned! Our code, dataset, and pretrain weights are coming soon." here (2024 November 26th), is finetuning not available at the moment? If so, how can I finetune this model for now?

Thank you.

[XuGW-Kevin]

No, you can finetune the model using exactly the method as Llama-3.2-11B-Vision-Instruct.
Most finetuning libraries support Llama-3.2-Vision, and we used https://github.com/Meta-Llama/llama-recipes.
You may follow the instructions here: https://github.com/meta-llama/llama-recipes/blob/main/recipes/quickstart/finetuning/finetune_vision_model.md

[Bleking]

No, you can finetune the model using exactly the method as Llama-3.2-11B-Vision-Instruct. Most finetuning libraries support Llama-3.2-Vision, and we used https://github.com/Meta-Llama/llama-recipes. You may follow the instructions here: https://github.com/meta-llama/llama-recipes/blob/main/recipes/quickstart/finetuning/finetune_vision_model.md

Thank you for that. But since I am not familiar with Llama-recipe, did you have to upload the custom dataset to Huggingface? I can try it as well but I am thinking of using the dataset in my local directory of my linux server.

I think I am suffering for the dataset setting since the settings for that are quite different from LLaVA-v1.5 and v1.6 on which I have been originally focusing on.

[XuGW-Kevin]

Yes, the dataset is available at https://huggingface.co/datasets/Xkev/LLaVA-CoT-100k.
The dataset format is actually very similar to LLaVA-v1.5 and v1.6.
If you need further help on the dataset format conversion, I can help with that.