Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Core][Frontend] Support Passing Multimodal Processor Kwargs #8657

Merged
merged 36 commits into from
Sep 23, 2024
Merged

[Core][Frontend] Support Passing Multimodal Processor Kwargs #8657

merged 36 commits into from
Sep 23, 2024

Commits on Sep 20, 2024

  1. Allow for processor kwarg overrides 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    550378b View commit details
    Browse the repository at this point in the history

  2. Pass processor through to partial 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    190606f View commit details
    Browse the repository at this point in the history

  3. Add default & processor kwarg override tests 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    b1ca041 View commit details
    Browse the repository at this point in the history

  4. Don't allow ctx or inputs as kwargs 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    195e31c View commit details
    Browse the repository at this point in the history

  5. Add kwarg override for processor to dummy data factories 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    1472d04 View commit details
    Browse the repository at this point in the history

  6. Add kwarg override forr processor to max token calc 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    f10601f View commit details
    Browse the repository at this point in the history

  7. Move kwarg only override func to utils 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    429097a View commit details
    Browse the repository at this point in the history

  8. Force processor kwargs to be keyword-only 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    159cfc2 View commit details
    Browse the repository at this point in the history

  9. Pass unfiltered processor kwargs to default mapper 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    af91930 View commit details
    Browse the repository at this point in the history

  10. Add hack for mapper preprocessor kwargs 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    9adad10 View commit details
    Browse the repository at this point in the history

  11. Simplify dummy data processor kwarg & add tests 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    9f7aed8 View commit details
    Browse the repository at this point in the history

  12. Add tests for max multimodal token kwarg overrides 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    ff59e44 View commit details
    Browse the repository at this point in the history

  13. Format registry 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    6b26454 View commit details
    Browse the repository at this point in the history

  14. Fix default mapper comparison 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    0e2d53d View commit details
    Browse the repository at this point in the history

  15. Move kwarg filtering into hf processor getter 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    5a3341b View commit details
    Browse the repository at this point in the history

  16. Enable processor_kwargs in video processor 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    3e1fe54 View commit details
    Browse the repository at this point in the history

  17. Add tests for mapper processor_kwargs 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    feccfd7 View commit details
    Browse the repository at this point in the history

  18. Update mapper not on multimodal processor kwargs 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    3ada64d View commit details
    Browse the repository at this point in the history

  19. processor kwarg test cleanup 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    58dcc63 View commit details
    Browse the repository at this point in the history

  20. Move context builder to test utils 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    1cee215 View commit details
    Browse the repository at this point in the history

  21. Use common context builder in processor kwarg tests 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    d5f9efa View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2024

  1. Update vllm/entrypoints/llm.py 

    Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
    @alex-jw-brooks @DarkLight1337
    alex-jw-brooks and DarkLight1337 authored Sep 22, 2024
    Configuration menu
    Copy the full SHA
    b5d434b View commit details
    Browse the repository at this point in the history

  2. Update vllm/inputs/registry.py 

    Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
    @alex-jw-brooks @DarkLight1337
    alex-jw-brooks and DarkLight1337 authored Sep 22, 2024
    Configuration menu
    Copy the full SHA
    a096301 View commit details
    Browse the repository at this point in the history

  3. Update vllm/inputs/registry.py 

    Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
    @alex-jw-brooks @DarkLight1337
    alex-jw-brooks and DarkLight1337 authored Sep 22, 2024
    Configuration menu
    Copy the full SHA
    79962e0 View commit details
    Browse the repository at this point in the history

  4. Update vllm/inputs/registry.py 

    Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
    @alex-jw-brooks @DarkLight1337
    alex-jw-brooks and DarkLight1337 authored Sep 22, 2024
    Configuration menu
    Copy the full SHA
    2cb1f72 View commit details
    Browse the repository at this point in the history

  5. Update vllm/inputs/registry.py 

    Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
    @alex-jw-brooks @DarkLight1337
    alex-jw-brooks and DarkLight1337 authored Sep 22, 2024
    Configuration menu
    Copy the full SHA
    37eb532 View commit details
    Browse the repository at this point in the history

  6. Update vllm/inputs/registry.py 

    Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
    @alex-jw-brooks @DarkLight1337
    alex-jw-brooks and DarkLight1337 authored Sep 22, 2024
    Configuration menu
    Copy the full SHA
    a4c7c3d View commit details
    Browse the repository at this point in the history

  7. Fix formatting 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    36dd2cb View commit details
    Browse the repository at this point in the history

  8. Rename processor kwargs to mm processor kwargs 

    Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
    @alex-jw-brooks
    alex-jw-brooks committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    f95c86f View commit details
    Browse the repository at this point in the history

  9. Update docstring

    @DarkLight1337
    DarkLight1337 authored Sep 22, 2024
    Configuration menu
    Copy the full SHA
    229604f View commit details
    Browse the repository at this point in the history

  10. Merge branch 'main' into support_processor_kwargs

    @DarkLight1337
    DarkLight1337 committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    2a48452 View commit details
    Browse the repository at this point in the history

  11. Try to fix CUDA reinitialization error

    @DarkLight1337
    DarkLight1337 committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    b732d72 View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. Consolidate processor loading

    @DarkLight1337
    DarkLight1337 committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    844524a View commit details
    Browse the repository at this point in the history

  2. Fix CUDA reinitialization error

    @DarkLight1337
    DarkLight1337 committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    2dd742b View commit details
    Browse the repository at this point in the history

  3. Simplify code

    @DarkLight1337
    DarkLight1337 committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    ebc1c02 View commit details
    Browse the repository at this point in the history

  4. Fix tests

    @DarkLight1337
    DarkLight1337 committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    a7f32f5 View commit details
    Browse the repository at this point in the history