Skip to content

[Bugfix] fix bf16 multimodal model hash #23623

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Aug 26, 2025
Merged

[Bugfix] fix bf16 multimodal model hash #23623

Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
4e64ec1
[Bugfix] fix bf16 multimodal model hash
yuekaizhang Aug 26, 2025
8c6b773
Update vllm/multimodal/hasher.py
yuekaizhang Aug 26, 2025
5646453
move to cpu
yuekaizhang Aug 26, 2025
9d39378
Update vllm/multimodal/hasher.py
yuekaizhang Aug 26, 2025
df66349
fix mypy error
yuekaizhang Aug 26, 2025
9471f50
Merge branch 'main' into bugfix
yuekaizhang Aug 26, 2025
8b35955
using 1d uint8
yuekaizhang Aug 26, 2025
6f7c652
Merge branch 'main' into bugfix
yuekaizhang Aug 26, 2025
4583d60
make sure contiguous
yuekaizhang Aug 26, 2025
34ae3c1
fix contiguous
yuekaizhang Aug 26, 2025
eb452fe
fix contiguous
yuekaizhang Aug 26, 2025
a69102c
Merge branch 'main' into bugfix
DarkLight1337 Aug 26, 2025
8a8dcf7
fix dtype
yuekaizhang Aug 26, 2025
67a0f70
fix shape
yuekaizhang Aug 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion vllm/multimodal/hasher.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,19 @@ def serialize_item(cls, obj: object) -> Union[bytes, memoryview]:
return cls.item_to_bytes(
"image", np.asarray(convert_image_mode(obj, "RGBA")))
if isinstance(obj, torch.Tensor):
return cls.item_to_bytes("tensor", obj.cpu().numpy())
tensor_obj: torch.Tensor = obj.cpu()
tensor_dtype = tensor_obj.dtype
if tensor_dtype == torch.bfloat16:
tensor_obj = tensor_obj.contiguous()
tensor_obj = tensor_obj.view(
(tensor_obj.numel(), )).view(torch.uint8)
return cls.item_to_bytes(
"tensor", {
"original_dtype": str(tensor_dtype),
"original_shape": tuple(tensor_obj.shape),
Comment on lines +54 to +55
Copy link
Contributor

@lgeiger lgeiger Aug 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for being late to the party. This is still the 1D shape of the uint8 tensor since tensor_obj is overwritten above. This should be obj.shape instead of tensor_obj.shape.

Could you extend the tests to verify the bfloat16 code path as well:

vllm/tests/multimodal/test_hasher.py

Lines 57 to 63 in 7ea22e4

def test_hash_collision_array_shape():
# The hash should be different though the data is the same when flattened
arr1 = np.zeros((5, 10, 20, 3))
arr2 = np.zeros((10, 20, 5, 3))
hasher = MultiModalHasher
assert hasher.hash_kwargs(data=arr1) != hasher.hash_kwargs(data=arr2)

Otherwise it would cause another CVE similar to #17378

Copy link
Member

@DarkLight1337 DarkLight1337 Aug 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, let me open another PR to fix

Copy link
Member

@DarkLight1337 DarkLight1337 Aug 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#23672

"data": tensor_obj.numpy()
})
return cls.item_to_bytes("tensor", tensor_obj.numpy())
if isinstance(obj, np.ndarray):
# If the array is non-contiguous, we need to copy it first
arr_data = obj.data if obj.flags.c_contiguous else obj.tobytes()
Expand Down