bug: Missing NGL Setting for Vision Models

[imtuyethan]

Jan version

0.5.10

Describe the Bug

https://discord.com/channels/1107178041848909847/1307915350406467664/1313079041821249556
The vision model configuration in model.json lacks an NGL (number of GPU layers) setting, which is needed to properly manage GPU resource allocation when running multiple models simultaneously (text model + vision/projector model). This causes uncertainty about proper GPU layer allocation between models.

Steps to Reproduce

1. Load a vision model with the following settings:

"settings": {
    "vision_model": true,
    "text_model": false,
    "ctx_len": 4096,
    "prompt_template": "\n### Instruction:\n{prompt}\n### Response:\n",
    "llama_model_path": "llava-v1.6-mistral-7b.Q4_K_M.gguf",
    "mmproj": "mmproj-model-f16.gguf"
}

2. Notice that NGL setting is missing, making it unclear how to allocate GPU layers when running multiple models

Screenshots / Logs

What is your OS?

• MacOS
• Windows
• Linux

[louis-jan]

Can check after supporting vision models pull.

[louis-jan]

On this one, it's more about supporting vision models which actually here #1493. There will be an action item to pull correct NGL from Chat model (not projector model). I tried it, and it really sped up. For now we will close this one and refer to the original epic as an action item, since now we temporarily fixed the listed models on the model hub.