Model: Granite docling + Idefics3 preprocessing (SmolVLM)

[gabe-l-hart]

NOTES

This PR replaces #16112 since that one was based on a branch in the core repo and I no longer have push access to the branch after #16113.

closes #16110

Description

This PR adds model support for https://huggingface.co/ibm-granite/granite-docling-258M. There are two key parts to the support:

1. Conversion support: Changes to convert_hf_to_gguf.py and the adjacent scripts/libraries to support the model
2. Overhaul of preprocessing support for idefics3

Details

The current code simply resizes the input to a square with padding. This is very different than the preprocessing implementation in transformers. From what I can tell, support for the more nuanced tile-based preprocessing was lost during the transition from llava to mtmd.

The logic in transformers is:

1. Resize the image so that the longest side is no larger than preprocessor_config.size.longest_edge (if do_resize here)
2. Resize and reshape the image so that both sides are an even multiple of image_size (if do_image_splitting here)
3. Create a N x M grid of images sized image_size x image_size where N = width / image_size and M = height / image_size
4. The input image is resized to image_size x image_size and added to the list of patches as the <global-img>
5. Each image in the grid is concatenated with <fake_token_around_image><row_R_col_C>{image tokens}
   1. If the image is the last in the row, a \n is added to the end
6. The global image is added with \n<fake_token_around_image><global-img>{image tokens}<fake_token_around_image>

Open Questions

In transformers, there are conditionals that have default values from preprocessor_config for do_resize and do_image_splitting. These can be overwritten with kwargs at invocation time. For both Granite Docling and SmolVLM, the defaults are true for both values, and the models give demonstrably bad results if the above tiling scheme is not followed. As such, I've opted to not make these configurable either in the hparams or as input to mtmd_encode. One alternate approach would be to decouple the preprocessing schemes from the models and give models default preprocessing schemes while allowing alternate schemes to be chosen at runtime. Given the poor results with the wrong preprocessing, though, I don't think this is worth the effort.

Testing

# Convert language model
 python convert_hf_to_gguf.py ~/models/ibm-granite/granite-docling-258M/

# Convert MMProj
python convert_hf_to_gguf.py ~/models/ibm-granite/granite-docling-258M/ --mmproj

# Run a sample
./bin/llama-mtmd-cli -m ~/models/ibm-granite/granite-docling-258M/granite-docling-258M-F16.gguf --image ~/Pictures/sample-doc.png --mmproj ~/models/ibm-granite/mmproj-granite-docling-258M -p "<__media__>Convert this page to markdown." --verbose -ngl 99

[gabe-l-hart]

Us and our long names 😞. I assume vertical alignment is worth preserving, but happy to not touch the other lines if preferred.

[gabe-l-hart]

I wasn't totally sure the right name for this one since it comes from preprocessor_config.size.longest_edge. This seemed appropriate since it's the size of the image for the first step of preprocessing, but I'm up for other names too (preproc_longest_edge?)

[gabe-l-hart]

This logic is similar to llava_uhd::get_slice_instructions, but it's different enough that I opted not to put this there for simplicity. I could certainly move it if that would be better though.

[gabe-l-hart]

I wasn't fully clear whether this was correct to limit to non-batch encoding, but since other tile-based models fall in this category, I figured it was correct. If we don't need this, we could also remove clip_is_idefics3.

[gabe-l-hart]

I tried removing this and things appear to work, so I've removed this part. I haven't tested robustly for concurrent inference, though.

[gabe-l-hart]

The overview image prefix is multiple tokens long for idefics3. It seemed likely that other models might have this for other delimiter tokens as well, so since they're all getting converted to std::vector<llama_token> to pass to add_text anyway, I figured it was more extensible to make all of these std::vector<llama_token>. It also avoids the need to check the sentinel value LLAMA_TOKEN_NULL.

[gabe-l-hart]

As written, there was no way to create the <row_R_col_C> delimiters since the row and col values need to be passed as inputs. I opted to use a basic printf style template here. One alternate approach would be to make this a nullable function handle that could be implemented with a lambda for individual models. That might be slightly more efficient to avoid the double snprintf below, but I'm not sure how that would trade off with the overhead of managing the function pointer and using std::string concatenation methods.

[CISC]

Generally LGTM, but I'll have to defer to @ngxson for mtmd.

[gabe-l-hart]

@ngxson A gentle nudge on reviewing this (without disrupting your vacation, of course!). The model has been sitting in the top-ten trending on HF (hitting #1 a few days ago), so there's a lot of community interest in getting this fully supported everywhere, especially for embedded inference with llama.cpp-based platforms.

[gabe-l-hart]

@ngxson Thanks for the review! I've addressed the comments.

Strangely, now I'm consistently seeing the image fail to terminate generation. Applying these same fixes to SmolVLM produces much better results than without and does not fail to terminate, so there seems to be something about granite-docling specifically causing this.

[ngxson]

Consider the changes don't touch token placements, it's quite surprising that we have such error. Did you also tried with top_k=1 ?

I haven't examined your code closely but maybe it's worth comparing its results againts calc_size_preserved_ratio, probably I missed something

[gabe-l-hart]

Ah, my comment was confusing. I tried at several points in history (before these review refactors, and before the most recent merge of master) and they all show the failure to stop, so I think it's something else. I was just surprised since I swear I wasn't seeing this when I was working on it a few weeks ago.

[ngxson]

After merging with latest master, the test model still runs well, so this should be good to merge

Any other problems can be addressed in follow-up PRs. Thanks @gabe-l-hart and the IBM team for the awesome model!

---

Full test result:

OK:   llama-mtmd-cli ggml-org/SmolVLM-500M-Instruct-GGUF:Q8_0
OK:   llama-mtmd-cli ggml-org/SmolVLM2-2.2B-Instruct-GGUF:Q4_K_M
OK:   llama-mtmd-cli ggml-org/SmolVLM2-500M-Video-Instruct-GGUF:Q8_0
OK:   llama-mtmd-cli ggml-org/gemma-3-4b-it-GGUF:Q4_K_M
OK:   llama-mtmd-cli THUDM/glm-edge-v-5b-gguf:Q4_K_M
OK:   llama-mtmd-cli second-state/Llava-v1.5-7B-GGUF:Q2_K
OK:   llama-mtmd-cli cjpais/llava-1.6-mistral-7b-gguf:Q3_K_M
OK:   llama-mtmd-cli ibm-research/granite-vision-3.2-2b-GGUF:Q4_K_M
OK:   llama-mtmd-cli second-state/MiniCPM-Llama3-V-2_5-GGUF:Q2_K
OK:   llama-mtmd-cli openbmb/MiniCPM-V-2_6-gguf:Q2_K
OK:   llama-mtmd-cli openbmb/MiniCPM-o-2_6-gguf:Q4_0
OK:   llama-mtmd-cli bartowski/Qwen2-VL-2B-Instruct-GGUF:Q4_K_M
OK:   llama-mtmd-cli ggml-org/Qwen2.5-VL-3B-Instruct-GGUF:Q4_K_M
OK:   llama-mtmd-cli ggml-org/InternVL2_5-1B-GGUF:Q8_0
OK:   llama-mtmd-cli ggml-org/InternVL3-1B-Instruct-GGUF:Q8_0
OK:   llama-mtmd-cli ggml-org/pixtral-12b-GGUF:Q4_K_M
OK:   llama-mtmd-cli ggml-org/Mistral-Small-3.1-24B-Instruct-2503-GGUF
OK:   llama-mtmd-cli ggml-org/Qwen2-VL-2B-Instruct-GGUF:Q4_K_M
OK:   llama-mtmd-cli ggml-org/Qwen2-VL-7B-Instruct-GGUF:Q4_K_M
OK:   llama-mtmd-cli ggml-org/Qwen2.5-VL-3B-Instruct-GGUF:Q4_K_M
OK:   llama-mtmd-cli ggml-org/Qwen2.5-VL-7B-Instruct-GGUF:Q4_K_M
OK:   llama-mtmd-cli ggml-org/InternVL3-8B-Instruct-GGUF:Q4_K_M
OK:   llama-mtmd-cli ggml-org/InternVL3-14B-Instruct-GGUF:Q4_K_M
OK:   llama-mtmd-cli ggml-org/Qwen2.5-VL-72B-Instruct-GGUF:Q4_K_M
OK:   llama-mtmd-cli ggml-org/Llama-4-Scout-17B-16E-Instruct-GGUF:IQ1_S

[gabe-l-hart]

Awesome, thanks for the review! I'll look into the stopping issue soon.

[gabe-l-hart]

Follow up created to address stopping issue: #16438