Local run instructions please

[ramkumarkoppu]

Hi,

Can you please provide instructions to run this model locally Linux/Mac computer?

[XuGW-Kevin]

Hi, to run this model locally on Linux/Mac computer, a good way is to convert the model to GGUF and use ollama.
However, llama.cpp does not support this feature.
It would be very hard for me to implement this, but it seems that the community has been working on this.

[XuGW-Kevin]

However, you can try running a quantized version of the model locally.
I found that the repository at unslothai supports quick quantization for using models in the llama-3.2-vision series, though I haven’t tried it myself. I find a demo here where you can run the model on a single T4 GPU.
If you’re willing to give it a try and encounter any issues, I’d be happy to work together to solve them. Thank you for your interest!