Add property-based testing for vLLM endpoints using an API defined by an OpenAPI 3.1 schema #16721

tarukumar · 2025-04-16T12:40:28Z

This script leverages the Schemathesis library to perform property-based testing on an API defined by an OpenAPI 3.1 schema. Integrated with pytest, it automates the testing process, ensuring the API adheres to its contract and identifying potential issues.

Note: Haven't added this test to the CI/CD workflow yet. Once everyone agrees, I can include it in a separate PR. Also, this test only covers stateless testing. If needed, I can create another PR for stateful testing, but that will require some additional work.

Logs from my local testing which Suggest that there needs to be work done on the api side :

warning: `uv run` is experimental and may change without warning
============================================================================================================ test session starts =============================================================================================================
platform linux -- Python 3.12.7, pytest-8.3.5, pluggy-1.5.0
rootdir: /home/takumar/Downloads
plugins: subtests-0.14.1, anyio-4.9.0, hypothesis-6.131.0, schemathesis-3.39.15
collected 1 item                                                                                                                                                                                                                             

test_new.py u,uuuu,,uu,,,,u,,,u.                                                                                                                                                                                                       [100%]

================================================================================================================== FAILURES ==================================================================================================================
_______________________________________________________________________________________ test_openapi_schema_with_fixture (verbose_name='GET /health') ________________________________________________________________________________________

    @wraps(test)
>   def test_function(*args: Any, **kwargs: Any) -> Any:

dsd/lib64/python3.12/site-packages/schemathesis/_hypothesis.py:83: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

case = Case(headers={'Content-Type': 'application/json'})

    @schema.parametrize()
    def test_openapi_schema_with_fixture(case):
        case.headers = {
            "Content-Type": "application/json",
        }
>       case.call_and_validate(verify=False)
E       schemathesis.exceptions.CheckFailed: 
E       
E       1. Missing Content-Type header
E       
E           The following media types are documented in the schema:
E           - `application/json`
E       
E       [200] OK:
E       
E           <EMPTY>
E       
E       Reproduce with: 
E       
E           curl -X GET -H 'Content-Type: application/json' --insecure http://127.0.0.1:8080/health
E       
E       Falsifying example: test_openapi_schema_with_fixture(
E           case=,
E       )

test_new.py:46: CheckFailed
----------------------------------------------------------------------------------------------------------- Captured stdout setup ------------------------------------------------------------------------------------------------------------
http://127.0.0.1:8080
------------------------------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------------------------------

________________________________________________________________________________________ test_openapi_schema_with_fixture (verbose_name='GET /ping') _________________________________________________________________________________________

    @wraps(test)
>   def test_function(*args: Any, **kwargs: Any) -> Any:

dsd/lib64/python3.12/site-packages/schemathesis/_hypothesis.py:83: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

case = Case(headers={'Content-Type': 'application/json'})

    @schema.parametrize()
    def test_openapi_schema_with_fixture(case):
        case.headers = {
            "Content-Type": "application/json",
        }
>       case.call_and_validate(verify=False)
E       schemathesis.exceptions.CheckFailed: 
E       
E       1. Missing Content-Type header
E       
E           The following media types are documented in the schema:
E           - `application/json`
E       
E       [200] OK:
E       
E           <EMPTY>
E       
E       Reproduce with: 
E       
E           curl -X GET -H 'Content-Type: application/json' --insecure http://127.0.0.1:8080/ping
E       
E       Falsifying example: test_openapi_schema_with_fixture(
E           case=,
E       )

test_new.py:46: CheckFailed
----------------------------------------------------------------------------------------------------------- Captured stdout setup ------------------------------------------------------------------------------------------------------------
http://127.0.0.1:8080
------------------------------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------------------------------

________________________________________________________________________________________ test_openapi_schema_with_fixture (verbose_name='POST /ping') ________________________________________________________________________________________

    @wraps(test)
>   def test_function(*args: Any, **kwargs: Any) -> Any:

dsd/lib64/python3.12/site-packages/schemathesis/_hypothesis.py:83: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

case = Case(headers={'Content-Type': 'application/json'})

    @schema.parametrize()
    def test_openapi_schema_with_fixture(case):
        case.headers = {
            "Content-Type": "application/json",
        }
>       case.call_and_validate(verify=False)
E       schemathesis.exceptions.CheckFailed: 
E       
E       1. Missing Content-Type header
E       
E           The following media types are documented in the schema:
E           - `application/json`
E       
E       [200] OK:
E       
E           <EMPTY>
E       
E       Reproduce with: 
E       
E           curl -X POST -H 'Content-Type: application/json' --insecure http://127.0.0.1:8080/ping
E       
E       Falsifying example: test_openapi_schema_with_fixture(
E           case=,
E       )

test_new.py:46: CheckFailed
----------------------------------------------------------------------------------------------------------- Captured stdout setup ------------------------------------------------------------------------------------------------------------
http://127.0.0.1:8080
------------------------------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------------------------------

______________________________________________________________________________________ test_openapi_schema_with_fixture (verbose_name='POST /tokenize') ______________________________________________________________________________________

    @wraps(test)
>   def test_function(*args: Any, **kwargs: Any) -> Any:

dsd/lib64/python3.12/site-packages/schemathesis/_hypothesis.py:83: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

case = Case(headers={'Content-Type': 'application/json'}, body={'messages': []})

    @schema.parametrize()
    def test_openapi_schema_with_fixture(case):
        case.headers = {
            "Content-Type": "application/json",
        }
>       case.call_and_validate(verify=False)
E       schemathesis.exceptions.CheckFailed: 
E       
E       1. Server error
E       
E       2. Undocumented HTTP status code
E       
E           Received: 500
E           Documented: 200, 422
E       
E       [500] Internal Server Error:
E       
E           `Internal Server Error`
E       
E       Reproduce with: 
E       
E           curl -X POST -H 'Content-Type: application/json' -d '{"messages": []}' --insecure http://127.0.0.1:8080/tokenize
E       
E       Falsifying example: test_openapi_schema_with_fixture(
E           case=,
E       )

test_new.py:46: CheckFailed
----------------------------------------------------------------------------------------------------------- Captured stdout setup ------------------------------------------------------------------------------------------------------------
http://127.0.0.1:8080
------------------------------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------------------------------

_____________________________________________________________________________________ test_openapi_schema_with_fixture (verbose_name='POST /detokenize') _____________________________________________________________________________________

    @wraps(test)
>   def test_function(*args: Any, **kwargs: Any) -> Any:

dsd/lib64/python3.12/site-packages/schemathesis/_hypothesis.py:83: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

case = Case(headers={'Content-Type': 'application/json'}, body={'tokens': [-1]})

    @schema.parametrize()
    def test_openapi_schema_with_fixture(case):
        case.headers = {
            "Content-Type": "application/json",
        }
>       case.call_and_validate(verify=False)
E       schemathesis.exceptions.CheckFailed: 
E       
E       1. Server error
E       
E       2. Undocumented HTTP status code
E       
E           Received: 500
E           Documented: 200, 422
E       
E       [500] Internal Server Error:
E       
E           `Internal Server Error`
E       
E       Reproduce with: 
E       
E           curl -X POST -H 'Content-Type: application/json' -d '{"tokens": [-1]}' --insecure http://127.0.0.1:8080/detokenize
E       
E       Falsifying example: test_openapi_schema_with_fixture(
E           case=,
E       )

test_new.py:46: CheckFailed
----------------------------------------------------------------------------------------------------------- Captured stdout setup ------------------------------------------------------------------------------------------------------------
http://127.0.0.1:8080
------------------------------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------------------------------

________________________________________________________________________________ test_openapi_schema_with_fixture (verbose_name='POST /v1/chat/completions') _________________________________________________________________________________

    @wraps(test)
>   def test_function(*args: Any, **kwargs: Any) -> Any:
E   schemathesis.exceptions.CheckFailed: Schemathesis found 2 distinct sets of failures.
E   ==================== 
E   
E   1. Server error
E   
E   2. Undocumented HTTP status code
E   
E       Received: 500
E       Documented: 200, 422
E   
E   [500] Internal Server Error:
E   
E       `Internal Server Error`
E   
E   Reproduce with: 
E   
E       curl -X POST -H 'Content-Type: application/json' -d '{"messages": []}' --insecure http://127.0.0.1:8080/v1/chat/completions
E   
E   ==================== 
E   
E   1. Undocumented HTTP status code
E   
E       Received: 400
E       Documented: 200, 422
E   
E   [400] Bad Request:
E   
E       `{"object":"error","message":"list object has no element 0","type":"BadRequestError","param":null,"code":400}`
E   
E   Reproduce with: 
E   
E       curl -X POST -H 'Content-Type: application/json' -d '{"messages": [{"role": "assistant"}]}' --insecure http://127.0.0.1:8080/v1/chat/completions

dsd/lib64/python3.12/site-packages/schemathesis/_hypothesis.py:83: CheckFailed
----------------------------------------------------------------------------------------------------------- Captured stdout setup ------------------------------------------------------------------------------------------------------------
http://127.0.0.1:8080
------------------------------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------------------------------

___________________________________________________________________________________ test_openapi_schema_with_fixture (verbose_name='POST /v1/completions') ___________________________________________________________________________________

    @wraps(test)
>   def test_function(*args: Any, **kwargs: Any) -> Any:
E   schemathesis.exceptions.CheckFailed: Schemathesis found 2 distinct sets of failures.
E   ==================== 
E   
E   1. Undocumented HTTP status code
E   
E       Received: 400
E       Documented: 200, 422
E   
E   [400] Bad Request:
E   
E       `{"object":"error","message":"please provide at least one prompt","type":"BadRequestError","param":null,"code":400}`
E   
E   Reproduce with: 
E   
E       curl -X POST -H 'Content-Type: application/json' -d '{"prompt": []}' --insecure http://127.0.0.1:8080/v1/completions
E   
E   ==================== 
E   
E   1. Server error
E   
E   2. Undocumented HTTP status code
E   
E       Received: 500
E       Documented: 200, 422
E   
E   [500] Internal Server Error:
E   
E       `Internal Server Error`
E   
E   Reproduce with: 
E   
E       curl -X POST -H 'Content-Type: application/json' -d '{"prompt": [-1]}' --insecure http://127.0.0.1:8080/v1/completions

dsd/lib64/python3.12/site-packages/schemathesis/_hypothesis.py:83: CheckFailed
----------------------------------------------------------------------------------------------------------- Captured stdout setup ------------------------------------------------------------------------------------------------------------
http://127.0.0.1:8080
------------------------------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------------------------------

______________________________________________________________________________ test_openapi_schema_with_fixture (verbose_name='POST /v1/audio/transcriptions') _______________________________________________________________________________

    @wraps(test)
>   def test_function(*args: Any, **kwargs: Any) -> Any:

dsd/lib64/python3.12/site-packages/schemathesis/_hypothesis.py:83: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

case = Case(headers={'Content-Type': 'application/json'}, body={'file': Binary(data=b'')})

    @schema.parametrize()
    def test_openapi_schema_with_fixture(case):
        case.headers = {
            "Content-Type": "application/json",
        }
>       case.call_and_validate(verify=False)
E       schemathesis.exceptions.CheckFailed: 
E       
E       1. Undocumented HTTP status code
E       
E           Received: 400
E           Documented: 200, 422
E       
E       [400] Bad Request:
E       
E           `{"object":"error","message":"[{'type': 'value_error', 'loc': ('body', 'file'), 'msg': \"Value error, Expected UploadFile, received: <class 'str'>\", 'input': \"b''\", 'ctx': {'error': ValueError(\"Expected UploadFile, received: <class 'str'>\")}}]","type":"BadRequestError","param":null,"code":400}`
E       
E       Reproduce with: 
E       
E           curl -X POST -H 'Content-Type: application/x-www-form-urlencoded' -d file=b%27%27 --insecure http://127.0.0.1:8080/v1/audio/transcriptions
E       
E       Falsifying example: test_openapi_schema_with_fixture(
E           case=,
E       )

test_new.py:46: CheckFailed
----------------------------------------------------------------------------------------------------------- Captured stdout setup ------------------------------------------------------------------------------------------------------------
http://127.0.0.1:8080
------------------------------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------------------------------

____________________________________________________________________________________ test_openapi_schema_with_fixture (verbose_name='POST /invocations') _____________________________________________________________________________________

    @wraps(test)
>   def test_function(*args: Any, **kwargs: Any) -> Any:

dsd/lib64/python3.12/site-packages/schemathesis/_hypothesis.py:83: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

case = Case(headers={'Content-Type': 'application/json'})

    @schema.parametrize()
    def test_openapi_schema_with_fixture(case):
        case.headers = {
            "Content-Type": "application/json",
        }
>       case.call_and_validate(verify=False)
E       schemathesis.exceptions.CheckFailed: 
E       
E       1. Server error
E       
E       2. Undocumented HTTP status code
E       
E           Received: 500
E           Documented: 200
E       
E       [500] Internal Server Error:
E       
E           `Internal Server Error`
E       
E       Reproduce with: 
E       
E           curl -X POST -H 'Content-Type: application/json' --insecure http://127.0.0.1:8080/invocations
E       
E       Falsifying example: test_openapi_schema_with_fixture(
E           case=,
E       )

test_new.py:46: CheckFailed
----------------------------------------------------------------------------------------------------------- Captured stdout setup ------------------------------------------------------------------------------------------------------------
http://127.0.0.1:8080
------------------------------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------------------------------

============================================================================================================== warnings summary ==============================================================================================================
dsd/lib64/python3.12/site-packages/schemathesis/generation/coverage.py:305
  /home/takumar/Downloads/dsd/lib64/python3.12/site-packages/schemathesis/generation/coverage.py:305: DeprecationWarning: jsonschema.exceptions.RefResolutionError is deprecated as of version 4.18.0. If you wish to catch potential reference resolution errors, directly catch referencing.exceptions.Unresolvable.
    ref_error: type[Exception] = jsonschema.RefResolutionError,

dsd/lib64/python3.12/site-packages/schemathesis/internal/deprecation.py:6
  /home/takumar/Downloads/dsd/lib64/python3.12/site-packages/schemathesis/internal/deprecation.py:6: DeprecationWarning: Argument `method` is deprecated and will be removed in Schemathesis 4.0. Use `include` and `exclude` methods instead.
    warnings.warn(

dsd/lib64/python3.12/site-packages/schemathesis/internal/deprecation.py:6
  /home/takumar/Downloads/dsd/lib64/python3.12/site-packages/schemathesis/internal/deprecation.py:6: DeprecationWarning: Argument `endpoint` is deprecated and will be removed in Schemathesis 4.0. Use `include` and `exclude` methods instead.
    warnings.warn(

dsd/lib64/python3.12/site-packages/schemathesis/internal/deprecation.py:6
  /home/takumar/Downloads/dsd/lib64/python3.12/site-packages/schemathesis/internal/deprecation.py:6: DeprecationWarning: Argument `tag` is deprecated and will be removed in Schemathesis 4.0. Use `include` and `exclude` methods instead.
    warnings.warn(

dsd/lib64/python3.12/site-packages/schemathesis/internal/deprecation.py:6
  /home/takumar/Downloads/dsd/lib64/python3.12/site-packages/schemathesis/internal/deprecation.py:6: DeprecationWarning: Argument `operation_id` is deprecated and will be removed in Schemathesis 4.0. Use `include` and `exclude` methods instead.
    warnings.warn(

test_new.py::test_openapi_schema_with_fixture
  /home/takumar/Downloads/dsd/lib64/python3.12/site-packages/schemathesis/specs/openapi/references.py:11: DeprecationWarning: jsonschema.exceptions.RefResolutionError is deprecated as of version 4.18.0. If you wish to catch potential reference resolution errors, directly catch referencing.exceptions.Unresolvable.
    from jsonschema.exceptions import RefResolutionError

test_new.py::test_openapi_schema_with_fixture
  /home/takumar/Downloads/dsd/lib64/python3.12/site-packages/schemathesis/specs/openapi/references.py:51: DeprecationWarning: jsonschema.RefResolver is deprecated as of v4.18.0, in favor of the https://github.com/python-jsonschema/referencing library, which provides more compliant referencing behavior as well as more flexible APIs for customization. A future release will remove RefResolver. Please file a feature request (on referencing) if you are missing an API for the kind of customization you need.
    class InliningResolver(jsonschema.RefResolver):

test_new.py::test_openapi_schema_with_fixture
  /home/takumar/Downloads/dsd/lib64/python3.12/site-packages/schemathesis/specs/openapi/schemas.py:89: DeprecationWarning: jsonschema.exceptions.RefResolutionError is deprecated as of version 4.18.0. If you wish to catch potential reference resolution errors, directly catch referencing.exceptions.Unresolvable.
    SCHEMA_PARSING_ERRORS = (KeyError, AttributeError, jsonschema.exceptions.RefResolutionError)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================================================================== short test summary info ===========================================================================================================
(verbose_name='GET /health') SUBFAIL test_new.py::test_openapi_schema_with_fixture[GET /health] - schemathesis.exceptions.CheckFailed: 
(verbose_name='GET /ping') SUBFAIL test_new.py::test_openapi_schema_with_fixture[GET /ping] - schemathesis.exceptions.CheckFailed: 
(verbose_name='POST /ping') SUBFAIL test_new.py::test_openapi_schema_with_fixture[POST /ping] - schemathesis.exceptions.CheckFailed: 
(verbose_name='POST /tokenize') SUBFAIL test_new.py::test_openapi_schema_with_fixture[POST /tokenize] - schemathesis.exceptions.CheckFailed: 
(verbose_name='POST /detokenize') SUBFAIL test_new.py::test_openapi_schema_with_fixture[POST /detokenize] - schemathesis.exceptions.CheckFailed: 
(verbose_name='POST /v1/chat/completions') SUBFAIL test_new.py::test_openapi_schema_with_fixture[POST /v1/chat/completions] - schemathesis.exceptions.CheckFailed: Schemathesis found 2 distinct sets of failures.
(verbose_name='POST /v1/completions') SUBFAIL test_new.py::test_openapi_schema_with_fixture[POST /v1/completions] - schemathesis.exceptions.CheckFailed: Schemathesis found 2 distinct sets of failures.
(verbose_name='POST /v1/audio/transcriptions') SUBFAIL test_new.py::test_openapi_schema_with_fixture[POST /v1/audio/transcriptions] - schemathesis.exceptions.CheckFailed: 
(verbose_name='POST /invocations') SUBFAIL test_new.py::test_openapi_schema_with_fixture[POST /invocations] - schemathesis.exceptions.CheckFailed: 
================================================================================== 9 failed, 1 passed, 8 warnings, 10 subtests passed in 690.65s (0:11:30) ===================================================================================

Signed-off-by: Tarun Kumar <takumar@redhat.com>

github-actions · 2025-04-16T12:40:40Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

Signed-off-by: Tarun Kumar <takumar@redhat.com>

tests/entrypoints/openai/test_openai_schema.py

Signed-off-by: Tarun Kumar <takumar@redhat.com>

tests/entrypoints/openai/test_openai_schema.py

Signed-off-by: Tarun Kumar <takumar@redhat.com>

tests/entrypoints/openai/test_openai_schema.py

DarkLight1337

Feel free to open GH issues regarding the failures when running this test locally so someone can pick them up and fix them.

tarukumar · 2025-04-17T10:55:27Z

Yes, that's the plan. I'll start logging the issues based on the testing I've done. Also, I think it would be beneficial to add this to the testing pipeline. If we agree on that, I can look into it as well.

DarkLight1337 · 2025-04-17T10:58:33Z

Sounds good to me!

Signed-off-by: Tarun Kumar <takumar@redhat.com>

mergify · 2025-04-17T14:45:55Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @tarukumar.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

DarkLight1337 · 2025-04-17T16:22:08Z

Please fix pre-commit

Signed-off-by: Nick Hill <nhill@redhat.com>

tarukumar · 2025-04-17T16:52:37Z

Done

… an OpenAPI 3.1 schema (vllm-project#16721) Signed-off-by: Tarun Kumar <takumar@redhat.com> Signed-off-by: Nick Hill <nhill@redhat.com> Co-authored-by: Nick Hill <nhill@redhat.com> Signed-off-by: Yang Wang <elainewy@meta.com>

… an OpenAPI 3.1 schema (vllm-project#16721) Signed-off-by: Tarun Kumar <takumar@redhat.com> Signed-off-by: Nick Hill <nhill@redhat.com> Co-authored-by: Nick Hill <nhill@redhat.com>

… an OpenAPI 3.1 schema (vllm-project#16721) Signed-off-by: Tarun Kumar <takumar@redhat.com> Signed-off-by: Nick Hill <nhill@redhat.com> Co-authored-by: Nick Hill <nhill@redhat.com> Signed-off-by: Agata Dobrzyniewicz <adobrzyniewicz@habana.ai>

… an OpenAPI 3.1 schema (vllm-project#16721) Signed-off-by: Tarun Kumar <takumar@redhat.com> Signed-off-by: Nick Hill <nhill@redhat.com> Co-authored-by: Nick Hill <nhill@redhat.com> Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>

Add tests for openai schema testing

8ffbdc8

Signed-off-by: Tarun Kumar <takumar@redhat.com>

tarukumar requested review from DarkLight1337, robertgshaw2-redhat and simon-mo as code owners April 16, 2025 12:40

mergify bot added the ci/build label Apr 16, 2025

tarukumar changed the title ~~Add tests for openai schema testing~~ Add property-based testing for vLLM endpoints using an API defined by an OpenAPI 3.1 schema Apr 16, 2025

fix precommit

a1ae7db

Signed-off-by: Tarun Kumar <takumar@redhat.com>

DarkLight1337 reviewed Apr 16, 2025

View reviewed changes

tests/entrypoints/openai/test_openai_schema.py Outdated Show resolved Hide resolved

replace model

d459a34

Signed-off-by: Tarun Kumar <takumar@redhat.com>

tarukumar requested a review from DarkLight1337 April 16, 2025 16:08

DarkLight1337 reviewed Apr 17, 2025

View reviewed changes

tests/entrypoints/openai/test_openai_schema.py Outdated Show resolved Hide resolved

DarkLight1337 reviewed Apr 17, 2025

View reviewed changes

tests/entrypoints/openai/test_openai_schema.py Outdated Show resolved Hide resolved

apply header to all genereted tc from schema

a2ceea6

Signed-off-by: Tarun Kumar <takumar@redhat.com>

tarukumar requested a review from DarkLight1337 April 17, 2025 10:20

DarkLight1337 reviewed Apr 17, 2025

View reviewed changes

tests/entrypoints/openai/test_openai_schema.py Show resolved Hide resolved

DarkLight1337 approved these changes Apr 17, 2025

View reviewed changes

DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Apr 17, 2025

ignore

9703dd5

Signed-off-by: Tarun Kumar <takumar@redhat.com>

DarkLight1337 enabled auto-merge (squash) April 17, 2025 13:36

mergify bot added the needs-rebase label Apr 17, 2025

Merge branch 'main' into test-openai

93e58fa

auto-merge was automatically disabled April 17, 2025 16:15
Head branch was pushed to by a user without write access

mergify bot removed the needs-rebase label Apr 17, 2025

njhill added 2 commits April 17, 2025 22:07

[V1][Perf] Faster incremental detokenization (vllm-project#15137)

59ea0f3

Signed-off-by: Nick Hill <nhill@redhat.com>

[V1][Perf] Faster incremental detokenization (vllm-project#15137)

779db30

Signed-off-by: Nick Hill <nhill@redhat.com>

vllm-bot merged commit e37073e into vllm-project:main Apr 18, 2025
63 of 69 checks passed

ckhordiasma mentioned this pull request May 14, 2025

nm vllm ent 0.8.5 sync red-hat-data-services/vllm#139

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Uh oh!

Add property-based testing for vLLM endpoints using an API defined by an OpenAPI 3.1 schema #16721

Add property-based testing for vLLM endpoints using an API defined by an OpenAPI 3.1 schema #16721

Uh oh!

tarukumar commented Apr 16, 2025 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Apr 16, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DarkLight1337 left a comment

Uh oh!

tarukumar commented Apr 17, 2025

Uh oh!

DarkLight1337 commented Apr 17, 2025

Uh oh!

mergify bot commented Apr 17, 2025

Uh oh!

DarkLight1337 commented Apr 17, 2025

Uh oh!

tarukumar commented Apr 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Uh oh!

Add property-based testing for vLLM endpoints using an API defined by an OpenAPI 3.1 schema #16721

Add property-based testing for vLLM endpoints using an API defined by an OpenAPI 3.1 schema #16721

Uh oh!

Conversation

tarukumar commented Apr 16, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 16, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DarkLight1337 left a comment

Choose a reason for hiding this comment

Uh oh!

tarukumar commented Apr 17, 2025

Uh oh!

DarkLight1337 commented Apr 17, 2025

Uh oh!

mergify bot commented Apr 17, 2025

Uh oh!

DarkLight1337 commented Apr 17, 2025

Uh oh!

tarukumar commented Apr 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tarukumar commented Apr 16, 2025 •

edited by github-actions bot

Loading