[Bugfix] fix bf16 multimodal model hash #23623

yuekaizhang · 2025-08-26T05:47:49Z

This PR fixed a bug when multimodal model and its embedding input is torch.bfloat16 tensor. Bfloat16 tensors can't be converted to numpy() directly.

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

github-actions · 2025-08-26T05:47:56Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

gemini-code-assist

Code Review

This pull request addresses a bug where hashing torch.bfloat16 tensors would fail. The proposed fix is to convert these tensors to a different data type before converting to a NumPy array. However, the chosen torch.float16 data type has a more limited range than bfloat16, which can lead to overflow and result in hash collisions for large numerical values. I've provided a critical comment with a suggestion to convert to torch.float32 instead, which avoids this issue. It would also be beneficial to add a new test case to verify the hashing of bfloat16 tensors and prevent future regressions.

vllm/multimodal/hasher.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

Isotr0py

Thanks for fixing!

vllm/multimodal/hasher.py

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

DarkLight1337

Thanks for fixing

ywang96

Thanks for the fix!

yuekaizhang · 2025-08-26T08:25:45Z

@DarkLight1337 Would you mind helping skip the failed pre-commit mypy CI tasks?

DarkLight1337 · 2025-08-26T08:27:42Z

You should fix the mypy error in this PR.

yuekaizhang · 2025-08-26T08:30:47Z

You should fix the mypy error in this PR.

I think the CI CD error existed before the PR. I could help to fix it if you could guide me.

Running mypy on 
Error: vllm/multimodal/hasher.py:47: error: "object" has no attribute "dtype"  [attr-defined]
Error: vllm/multimodal/hasher.py:48: error: "object" has no attribute "to"  [attr-defined]
Error: vllm/multimodal/hasher.py:49: error: "object" has no attribute "numpy"  [attr-defined]

vllm/multimodal/hasher.py

Co-authored-by: Roger Wang <hey@rogerw.io> Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

vllm/multimodal/hasher.py

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

vllm/multimodal/hasher.py

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

DarkLight1337 · 2025-08-26T10:39:49Z

Should be good to go now, thanks

vllm/multimodal/hasher.py

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

vllm/multimodal/hasher.py

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

DarkLight1337

I think this should work now

lgeiger · 2025-08-26T16:08:20Z

vllm/multimodal/hasher.py

+                        "original_dtype": str(tensor_dtype),
+                        "original_shape": tuple(tensor_obj.shape),


Sorry for being late to the party. This is still the 1D shape of the uint8 tensor since tensor_obj is overwritten above. This should be obj.shape instead of tensor_obj.shape.

Could you extend the tests to verify the bfloat16 code path as well:

vllm/tests/multimodal/test_hasher.py

Lines 57 to 63 in 7ea22e4

def test_hash_collision_array_shape():

# The hash should be different though the data is the same when flattened

arr1 = np.zeros((5, 10, 20, 3))

arr2 = np.zeros((10, 20, 5, 3))

hasher = MultiModalHasher

assert hasher.hash_kwargs(data=arr1) != hasher.hash_kwargs(data=arr2)

Otherwise it would cause another CVE similar to #17378

Oops, let me open another PR to fix

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Roger Wang <hey@rogerw.io> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk> Signed-off-by: tc-mb <caitianchi@modelbest.cn>

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Roger Wang <hey@rogerw.io> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Roger Wang <hey@rogerw.io> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk> Signed-off-by: Xiao Yu <xiao.yu@amd.com>

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Roger Wang <hey@rogerw.io> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

[Bugfix] fix bf16 multimodal model hash

4e64ec1

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

yuekaizhang requested review from DarkLight1337 and ywang96 as code owners August 26, 2025 05:47

mergify bot added the multi-modality Related to multi-modality (#4194) label Aug 26, 2025

gemini-code-assist bot reviewed Aug 26, 2025

View reviewed changes

vllm/multimodal/hasher.py Outdated Show resolved Hide resolved

Update vllm/multimodal/hasher.py

8c6b773

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

Isotr0py approved these changes Aug 26, 2025

View reviewed changes

Isotr0py added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 26, 2025

DarkLight1337 reviewed Aug 26, 2025

View reviewed changes

vllm/multimodal/hasher.py Outdated Show resolved Hide resolved

move to cpu

5646453

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

DarkLight1337 approved these changes Aug 26, 2025

View reviewed changes

DarkLight1337 enabled auto-merge (squash) August 26, 2025 07:25

ywang96 approved these changes Aug 26, 2025

View reviewed changes

DarkLight1337 reviewed Aug 26, 2025

View reviewed changes

vllm/multimodal/hasher.py Outdated Show resolved Hide resolved

ywang96 reviewed Aug 26, 2025

View reviewed changes

vllm/multimodal/hasher.py Outdated Show resolved Hide resolved

Update vllm/multimodal/hasher.py

9d39378

Co-authored-by: Roger Wang <hey@rogerw.io> Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

auto-merge was automatically disabled August 26, 2025 08:36
Head branch was pushed to by a user without write access

yuekaizhang added 2 commits August 26, 2025 17:46

fix mypy error

df66349

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

Merge branch 'main' into bugfix

9471f50

DarkLight1337 enabled auto-merge (squash) August 26, 2025 09:53

lgeiger reviewed Aug 26, 2025

View reviewed changes

vllm/multimodal/hasher.py Outdated Show resolved Hide resolved

using 1d uint8

8b35955

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

auto-merge was automatically disabled August 26, 2025 10:17
Head branch was pushed to by a user without write access

yuekaizhang added 2 commits August 26, 2025 18:18

Merge branch 'main' into bugfix

6f7c652

make sure contiguous

4583d60

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

DarkLight1337 reviewed Aug 26, 2025

View reviewed changes

vllm/multimodal/hasher.py Show resolved Hide resolved

yuekaizhang added 2 commits August 26, 2025 18:32

fix contiguous

34ae3c1

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

fix contiguous

eb452fe

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

DarkLight1337 enabled auto-merge (squash) August 26, 2025 10:39

Merge branch 'main' into bugfix

a69102c

lgeiger reviewed Aug 26, 2025

View reviewed changes

vllm/multimodal/hasher.py Show resolved Hide resolved

fix dtype

8a8dcf7

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

auto-merge was automatically disabled August 26, 2025 11:18
Head branch was pushed to by a user without write access

lgeiger reviewed Aug 26, 2025

View reviewed changes

vllm/multimodal/hasher.py Outdated Show resolved Hide resolved

fix shape

67a0f70

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

DarkLight1337 reviewed Aug 26, 2025

View reviewed changes

DarkLight1337 merged commit 513298f into vllm-project:main Aug 26, 2025
36 checks passed

lgeiger reviewed Aug 26, 2025

View reviewed changes

		"original_dtype": str(tensor_dtype),
		"original_shape": tuple(tensor_obj.shape),

	def test_hash_collision_array_shape():
	# The hash should be different though the data is the same when flattened
	arr1 = np.zeros((5, 10, 20, 3))
	arr2 = np.zeros((10, 20, 5, 3))

	hasher = MultiModalHasher
	assert hasher.hash_kwargs(data=arr1) != hasher.hash_kwargs(data=arr2)

Uh oh!

[Bugfix] fix bf16 multimodal model hash #23623

[Bugfix] fix bf16 multimodal model hash #23623

Uh oh!

Conversation

yuekaizhang commented Aug 26, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 26, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Isotr0py left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

DarkLight1337 left a comment

Choose a reason for hiding this comment

Uh oh!

ywang96 left a comment

Choose a reason for hiding this comment

Uh oh!

yuekaizhang commented Aug 26, 2025

Uh oh!

DarkLight1337 commented Aug 26, 2025

Uh oh!

yuekaizhang commented Aug 26, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DarkLight1337 commented Aug 26, 2025

Uh oh!

Uh oh!

Uh oh!

DarkLight1337 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lgeiger Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

DarkLight1337 Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

DarkLight1337 Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yuekaizhang commented Aug 26, 2025 •

edited by github-actions bot

Loading