LLaVA-OneVision for single-image, multi-image, and video

[KevinH48264]

Thank you for the great work here!

LLaVA-OneVision seems to be the latest model from the LLaVA family of models that performs well out of open source LMMs. While it's still early, raising it here as others in the community may start to look at it.

https://llava-vl.github.io/blog/2024-08-05-llava-onevision/

[zjysteven]

Thanks for raising this. We've also noticed it, and including it into lmms-finetune is part of our recent plan.

[zjysteven]

We will wait a bit until the model implementation is merged into the main branch of transformers.
https://huggingface.co/llava-hf/llava-onevision-qwen2-0.5b-ov-hf/discussions/1
huggingface/transformers#32673

[btekin]

Are there any updates on this? It seems LLaVA-Onevision got merged into the main branch of transformers now: huggingface/transformers#32673

[zjysteven]

We will include it once we are free, probably in 2 - 3 weeks.