Code

Add Granite Vision Support#11794

Code

Open

alex-jw-brooks wants to merge 37 commits intoggml-org:masterggml-org/llama.cpp:masterfrom alex-jw-brooks:granite_visionalex-jw-brooks/llama.cpp:granite_visionCopy head branch name to clipboard

+239-41

Commits

Commits on Feb 20, 2025

• Add super wip scripts for multimodal granite gguf

  Show description for 88e3d9d
  alex-jw-brooks

  committed
  88e3d9d

  Copy full SHA for 88e3d9d

• Add example for converting mmgranite to gguf

  Show description for d5da8c9
  alex-jw-brooks

  committed
  d5da8c9

  Copy full SHA for d5da8c9

• remove hardcoded path

  Show description for af6585d
  alex-jw-brooks

  committed
  af6585d

  Copy full SHA for af6585d

• Add vision feature layer to gguf params

  Show description for ab815a5
  alex-jw-brooks

  committed
  ab815a5

  Copy full SHA for ab815a5

• Clean up llava surgery and remove name substitution hacks

  Show description for 26ce7f5
  alex-jw-brooks

  committed
  26ce7f5

  Copy full SHA for 26ce7f5

• Add transformers llava next tensor name mapping

  Show description for 98bb2ff
  alex-jw-brooks

  committed
  98bb2ff

  Copy full SHA for 98bb2ff

• Make siglip / openclip mutuall exclusive

  Show description for 0fc506c
  alex-jw-brooks

  committed
  0fc506c

  Copy full SHA for 0fc506c

• Fix projector linear substitution

  Show description for 6508699
  alex-jw-brooks

  committed
  6508699

  Copy full SHA for 6508699

• Fix linear 2 substitution index

  Show description for 7f036c9
  alex-jw-brooks

  committed
  7f036c9

  Copy full SHA for 7f036c9

• Increase max flattened gridpoints to 64

  Show description for b43a0de
  alex-jw-brooks

  committed
  b43a0de

  Copy full SHA for b43a0de

• Fix hardcoded concat for multiple feature layers

  Show description for 097acd5
  alex-jw-brooks

  committed
  097acd5

  Copy full SHA for 097acd5

• Pull vision feature layers out of gguf keys

  Show description for 9a97e9b
  alex-jw-brooks

  committed
  9a97e9b

  Copy full SHA for 9a97e9b

• fix num gridpoints and use all layers

  Show description for b05613b
  alex-jw-brooks

  committed
  b05613b

  Copy full SHA for b05613b

• Avoid dropping last image encoder layer in llava models

  Show description for 44645ca
  alex-jw-brooks

  committed
  44645ca

  Copy full SHA for 44645ca

• Use 10 for max number of patches

  Show description for a37b176
  alex-jw-brooks

  committed
  a37b176

  Copy full SHA for a37b176

• Standardize vision feature layers

  Show description for 56e6364
  alex-jw-brooks

  committed
  56e6364

  Copy full SHA for 56e6364

• Cleanup logs

  Show description for 45ab64b
  alex-jw-brooks

  committed
  45ab64b

  Copy full SHA for 45ab64b

• Update comment for vision feature layer init

  Show description for 4990cfa
  alex-jw-brooks

  committed
  4990cfa

  Copy full SHA for 4990cfa

• Update notes for alternative to legacy llm conversion script

  Show description for 473606c
  alex-jw-brooks

  committed
  473606c

  Copy full SHA for 473606c

• Fix notes rendering

  Show description for 600ab1f
  alex-jw-brooks

  committed
  600ab1f

  Copy full SHA for 600ab1f

• Add v prefix to vision feature layer log

  Show description for 8b6db6c
  alex-jw-brooks

  committed
  8b6db6c

  Copy full SHA for 8b6db6c

• Use current defaults for feature layer

  Show description for 48a941d
  alex-jw-brooks

  committed
  48a941d

  Copy full SHA for 48a941d

• Use constant for max gridpoints / feat layers, style fixes

  Show description for b1be52a
  alex-jw-brooks

  committed
  b1be52a

  Copy full SHA for b1be52a

• clarify non-negative feature layers

  Show description for c35956d
  alex-jw-brooks

  committed
  c35956d

  Copy full SHA for c35956d

• Remove CLIP_API from func signature

  Show description for 71a67b0
  alex-jw-brooks

  committed
  71a67b0

  Copy full SHA for 71a67b0

• USE MAX_IMAGE_FEATURE_LAYERS const in layer calc

  Show description for ba0ef5d
  alex-jw-brooks

  committed
  ba0ef5d

  Copy full SHA for ba0ef5d

• Clarify feature layers are non negative ints and not uint

  Show description for b88fe16
  alex-jw-brooks

  committed
  b88fe16

  Copy full SHA for b88fe16

• Fix condition for reading feature layers

  Show description for f825e6b
  alex-jw-brooks

  committed
  f825e6b

  Copy full SHA for f825e6b

• pop last llava layer when feature layers are unset

  Show description for 6ee4be1
  alex-jw-brooks

  committed
  6ee4be1

  Copy full SHA for 6ee4be1

• Fix unset vision layer 0

  Show description for 7566532
  alex-jw-brooks

  committed
  7566532

  Copy full SHA for 7566532

• Update examples/llava/clip.cpp

  Show description for 23ece06
  

  alex-jw-brooks

  and

  ngxson

  committed
  23ece06

  Copy full SHA for 23ece06

• Reenable assertion for out of bounds get_rows

  Show description for 16a95d6
  alex-jw-brooks

  committed
  16a95d6

  Copy full SHA for 16a95d6

Commits on Feb 21, 2025

• Use std vector for gridpoints and feature layers

  Show description for 11149b6
  alex-jw-brooks

  committed
  11149b6

  Copy full SHA for 11149b6

• Caculate max feature layer at load time

  Show description for 92952d6
  alex-jw-brooks

  committed
  92952d6

  Copy full SHA for 92952d6

• Include base patch for granite vision allocation

  Show description for 49ef7c1
  alex-jw-brooks

  committed
  49ef7c1

  Copy full SHA for 49ef7c1

• Fix trailing whitespace

  Show description for a36e120
  alex-jw-brooks

  committed
  a36e120

  Copy full SHA for a36e120

• Add max num patches = 10 back for minicpmv

  Show description for ed4d55d
  alex-jw-brooks

  committed
  ed4d55d

  Copy full SHA for ed4d55d