[CI Failure]: Classification test failure for Qwen2.5-1.5B-apeach model in half precision #21277
Description
Name of failing test
FAILED models/test_transformers.py::test_classify[half-jason9693/Qwen2.5-1.5B-apeach] - AssertionError: assert False
Basic information
- Flaky test
- Can reproduce locally
- Caused by external libraries (e.g. bug in
transformers)
🧪 Describe the failing test
During test execution for the jason9693/Qwen2.5-1.5B-apeach model using half-precision dtype, the classification test test_classify failed due to output mismatch between vLLM and Hugging Face implementations.
[2025-07-20T11:45:11Z] > assert torch.allclose(hf_output, vllm_output,
--
| [2025-07-20T11:45:11Z] 1e-3 if dtype == "float" else 1e-2)
| [2025-07-20T11:45:11Z] E AssertionError: assert False
| [2025-07-20T11:45:11Z] E + where False = <built-in method allclose of type object at 0x7f7c776d8fa0>(tensor([0.0413, 0.9585]), tensor([0.0409, 0.9591]), 0.01)
| [2025-07-20T11:45:11Z] E + where <built-in method allclose of type object at 0x7f7c776d8fa0> = <module 'torch' from '/usr/local/lib/python3.12/dist-packages/torch/__init__.py'>.allclose
If it returns True when manually run locally, that's indeed quite strange.
import torch
tensor1 = torch.tensor([0.0413, 0.9585], dtype=torch.half)
tensor2 = torch.tensor([0.0409, 0.9591], dtype=torch.half)
print(torch.allclose(tensor1, tensor2,0.01)) # output true
📝 History of failing test
Ref to https://buildkite.com/vllm/ci/builds/24465#01982797-9656-4f51-883b-f284878d6cdf
Ref to #21243
CC List.
No response