Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature]: Support image embeddings as input for qwen2vl #8857

Closed
1 task done
whyiug opened this issue Sep 26, 2024 · 0 comments · Fixed by #8856
Closed
1 task done

[Feature]: Support image embeddings as input for qwen2vl #8857

whyiug opened this issue Sep 26, 2024 · 0 comments · Fixed by #8856

Comments

@whyiug
Copy link
Contributor

whyiug commented Sep 26, 2024

🚀 The feature, motivation and pitch

Most multimodal models support input image embeddings. see previous pr: #6613
IMO there's no reason not to support qwen2vl.

When I was about to add this feature to qwen2vl. Unfortunately, I've run into some difficulties.
For example, I can't just rely on image embedding to generate new prompt_token_ids without the original image. See here

height, width = get_image_size(image, channel_dim=input_data_format)

And here, if we just return image embeds, it will occur an error. AssertionError: mrope embedding type requires multi-modal input mapper returns 'image_grid_thw' or 'video_grid_thw'.

Might we need to passthrough more parameters for qwen2vl? please me give some tips.
here is my draft code: #8856

Alternatives

No response

Additional context

No response

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant