feat(model): add qwen3vl #12665

mxyng · 2025-10-16T19:26:56Z

No description provided.

dhiltgen · 2025-10-16T21:28:44Z

Can you add it to the list of image models we test here?

ml/backend/ggml/ggml.go

ml/nn/rope/rope.go

model/models/qwen3vl/model.go

model/models/qwen25vl/model_vision.go

model/models/qwen3vl/imageprocessor.go

pdevine · 2025-10-17T01:29:15Z

model/models/qwen3vl/model_vision.go

+}
+
+func (m *VisionModel) positions(ctx ml.Context, grid *Grid) (_, _ ml.Tensor) {
+	indices := ctx.Input().FromIntSlice(slices.Collect(func(yield func(int32) bool) {


Why not just return the slice here? Aren't you yielding and then immediately collecting?

this is cleaner. the alternative is to build up a slice instead of simply generating a slice

pdevine · 2025-10-17T01:32:56Z

model/models/qwen3vl/model_vision.go

+
+	halfDim := m.headDim() / 2
+	maxGrid := max(grid.Height, grid.Width)
+	frequencies := ctx.Input().FromFloatSlice(slices.Collect(func(yield func(float32) bool) {


Same comment here. Why not just return the slice instead the complexity of the iterator?

pdevine · 2025-10-17T01:34:28Z

model/models/qwen3vl/model_vision.go

+			spatialMergeSize:  int(c.Uint("vision.spatial_merge_size", 2)),
+			temporalPatchSize: int(c.Uint("vision.temporal_patch_size", 2)),
+			gridPerSide:       int(math.Sqrt(float64(c.Uint("vision.num_positional_embeddings", 2304)))),
+			mropeSections: slices.Collect(func(yield func(int) bool) {


Same comment as above.

jmorganca · 2025-10-28T16:58:54Z

model/models/qwen3vl/model_vision.go

+func makeSlice2D[T int32 | float32](n0, n1 int) iter.Seq[[]T] {
+	return func(yield func([]T) bool) {
+		for range n0 {
+			if !yield(make([]T, n1)) {
+				return
+			}
+		}
+	}
+}


are iterators required here? (For readability)

jmorganca

LGTM, @pdevine should take another look (and has some outstanding comments)

jessegross · 2025-10-28T18:07:55Z

runner/ollamarunner/cache.go

+	var discard int32
+	for discard < max(targetFree-currentFree, 0) {
+		if sameBatch := inputs[numKeep+discard].SameBatch; sameBatch > 0 {
+			discard += int32(sameBatch)


SameBatch is the number of tokens following the current one that need to be in the same batch so I believe that this should be discard += 1 + int32(sameBatch). You actually should not need to special case it - if SameBatch is 0 then it the same as the current non-SameBatch case.

The behavior of this loop is a little bit different from how we do truncation in NewSequence, which is SameBatch aware. That one will keep extending discard is there are overlapping SameBatch. That scenario is sort of undefined behavior but it's better to be consistent about it.

SameBatch is the number of tokens following the current one

that's not how it's being used right now. models are setting SameBatch to include the token setting SameBatch

It looks like some models probably use SameBatch that way but others use the original definition. Regardless, the runner has always executed batches as described above, so that's how these models are being run. It doesn't help things for shifting to have a different interpretation from the rest of the runner.

pdevine · 2025-10-28T17:45:13Z

model/models/qwen3vl/model.go

+// PostTokenize arranges Qwen 3 VL's inputs for the forward pass
+func (m *Model) PostTokenize(inputs []*input.Input) ([]*input.Input, error) {
+	m.positionCache = m.positionCache[:0]
+	return slices.Collect(func(yield func(*input.Input) bool) {


I'm don't think the Collect() / yield iterator pattern is adding anything here

model/models/qwen3vl/model.go

pdevine · 2025-10-28T18:16:48Z

model/models/qwen3vl/model_vision.go

+}
+
+func (m *VisionPositionEmbedding) Forward(ctx ml.Context, hiddenStates ml.Tensor, grid *Grid, opts VisionOptions) ml.Tensor {
+	indexSlice := slices.Collect(makeSlice2D[int32](4, grid.Height*grid.Width))


This would be easier to read as:

indexSlice := make([][]int32, 4) weightSlice := make([][]float32, 4)

and then just appending the ints/float32s inside of the nested loop below.

runner/ollamarunner/cache_test.go

jessegross

The runner/cache/GGML changes look good to me. I didn't review anything specific to the model itself.

pdevine

Ship it!

mxyng changed the base branch from main to mxyng/convert October 16, 2025 19:27

mxyng force-pushed the mxyng/convert branch from 4a55568 to 90dab29 Compare October 16, 2025 19:57

mxyng force-pushed the mxyng/qwen3vl branch from bb2fb03 to 54b87f3 Compare October 16, 2025 19:57

mxyng force-pushed the mxyng/convert branch from 90dab29 to 31219ed Compare October 16, 2025 20:03

mxyng force-pushed the mxyng/qwen3vl branch 2 times, most recently from eddb168 to cc6ed87 Compare October 16, 2025 20:24

jmorganca requested review from dhiltgen and pdevine October 16, 2025 21:11

mxyng force-pushed the mxyng/convert branch from 31219ed to 32d399f Compare October 16, 2025 22:18

mxyng force-pushed the mxyng/qwen3vl branch from cc6ed87 to 79dbf8f Compare October 16, 2025 22:19

mxyng force-pushed the mxyng/convert branch 2 times, most recently from 2bc23ea to 5c51e3e Compare October 16, 2025 22:59

mxyng force-pushed the mxyng/qwen3vl branch from 79dbf8f to 15589ea Compare October 16, 2025 23:07

mxyng force-pushed the mxyng/convert branch 2 times, most recently from e59fb68 to 13b5d3a Compare October 16, 2025 23:58

pdevine reviewed Oct 17, 2025

View reviewed changes

mxyng force-pushed the mxyng/qwen3vl branch from 15589ea to 53ce602 Compare October 20, 2025 20:12

mxyng force-pushed the mxyng/convert branch 4 times, most recently from 2bd4b6e to 2b26dc7 Compare October 20, 2025 21:19

mxyng force-pushed the mxyng/qwen3vl branch from 53ce602 to 34e4e34 Compare October 20, 2025 21:20

mxyng changed the base branch from mxyng/convert to mxyng/server-tests October 20, 2025 21:21

mxyng force-pushed the mxyng/server-tests branch from b5535ec to 05bc209 Compare October 20, 2025 23:42

mxyng force-pushed the mxyng/qwen3vl branch 5 times, most recently from b51c8bb to c9b37ec Compare October 22, 2025 18:47

mxyng force-pushed the mxyng/qwen3vl branch from 5491083 to 354d7cc Compare October 27, 2025 22:00

mxyng changed the base branch from mxyng/server-tests to main October 27, 2025 22:35

mxyng force-pushed the mxyng/qwen3vl branch 3 times, most recently from 7d5d232 to 9d60e9b Compare October 28, 2025 03:27

jmorganca reviewed Oct 28, 2025

View reviewed changes

jmorganca approved these changes Oct 28, 2025

View reviewed changes

jessegross reviewed Oct 28, 2025

View reviewed changes

mxyng force-pushed the mxyng/qwen3vl branch from 8edf98f to 18d1375 Compare October 28, 2025 18:22

pdevine reviewed Oct 28, 2025

View reviewed changes

mxyng force-pushed the mxyng/qwen3vl branch 6 times, most recently from 6be5624 to 77ea19a Compare October 28, 2025 20:42

jessegross approved these changes Oct 28, 2025

View reviewed changes

mxyng force-pushed the mxyng/qwen3vl branch 2 times, most recently from d655b0c to 26a8bb7 Compare October 28, 2025 22:47

mxyng added 2 commits October 28, 2025 17:16

ml(ggml): conv3d

bba26ca

ml(ggml): infer shape

ad9fec3

mxyng force-pushed the mxyng/qwen3vl branch from 26a8bb7 to 2731559 Compare October 29, 2025 00:16

mxyng added 4 commits October 28, 2025 17:21

feat(model): add qwen3vl

d47cd10

remove extra textmodel fields

e84076e

discard same batch

57586dd

integration

9fe6095

mxyng force-pushed the mxyng/qwen3vl branch from 2731559 to 9fe6095 Compare October 29, 2025 00:21

pdevine approved these changes Oct 29, 2025

View reviewed changes

mxyng merged commit 7d25b9e into main Oct 29, 2025
9 checks passed

mxyng deleted the mxyng/qwen3vl branch October 29, 2025 00:39

feat(model): add qwen3vl #12665

feat(model): add qwen3vl #12665

Conversation

mxyng commented Oct 16, 2025

Uh oh!

dhiltgen commented Oct 16, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jmorganca left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jessegross left a comment

Choose a reason for hiding this comment

Uh oh!

pdevine left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants