[CI/Build] Disable flaky test_structured_output tests #24404
Conversation
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request disables several flaky tests in test_structured_output to improve CI stability, which is a reasonable temporary measure. The associated tracking issue is correctly referenced. However, I've identified a latent bug in one of the commented-out test configurations that should be fixed to prevent future errors.
| #FIXME: This tests are flaky on CI thus disabled. Tracking in Issue #24402 | ||
| # ("mistralai/Ministral-8B-Instruct-2410", "outlines", "auto", None), | ||
| # ("mistralai/Ministral-8B-Instruct-2410", "outlines", "mistral", None), | ||
| #("Qwen/Qwen2.5-1.5B-Instruct", "guidance", "auto"), |
There was a problem hiding this comment.
This commented-out test case tuple is malformed. It has 3 elements, but the pytest.mark.parametrize for test_structured_output expects 4 elements (model_name, guided_decoding_backend, tokenizer_mode, speculative_config). If this test is uncommented in the future, it will cause a ValueError during test collection. Please add None as the fourth element to represent the speculative_config.
| #("Qwen/Qwen2.5-1.5B-Instruct", "guidance", "auto"), | |
| #("Qwen/Qwen2.5-1.5B-Instruct", "guidance", "auto", None), |
22quinn
added
ready
…4404) Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
…4404) Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
…4404) Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
…4404) Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
…4404) Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
…4404) Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
2 participants
Purpose
Mitigate #24402
Test Plan
CI
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.