Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Text2ImagePipeline to llm bench #1129

Merged
merged 3 commits into from
Nov 5, 2024
Merged

Add Text2ImagePipeline to llm bench #1129

merged 3 commits into from
Nov 5, 2024
+159 −26

Conversation

sbalandi
Copy link
Contributor

@sbalandi sbalandi commented Nov 1, 2024 •
edited by andrei-kochin
Loading

Ticket: CVS-156418

@github-actions github-actions bot added category: llm_bench Label for tool/llm_bench folder category: sampling Sampling / Decoding algorithms labels Nov 1, 2024
@sbalandi sbalandi force-pushed the image_gen branch 3 times, most recently from 5ea79fe to a0c7a8b Compare November 1, 2024 17:20
@ilya-lavrenov ilya-lavrenov added this to the 2024.5 milestone Nov 1, 2024
@github-actions github-actions bot added category: WWB PR changes WWB category: text to image Text 2 image pipeline category: Python API Python API for GenAI labels Nov 1, 2024
@sbalandi sbalandi requested a review from eaidova November 1, 2024 20:12
@github-actions github-actions bot removed the category: WWB PR changes WWB label Nov 1, 2024
@sbalandi sbalandi marked this pull request as ready for review November 1, 2024 20:21
ilya-lavrenov
ilya-lavrenov reviewed Nov 1, 2024
View reviewed changes
tools/llm_bench/benchmark.py Outdated Show resolved Hide resolved
tools/llm_bench/llm_bench_utils/ov_utils.py Outdated Show resolved Hide resolved
src/python/py_image_generation_pipelines.cpp Outdated Show resolved Hide resolved
@github-actions github-actions bot removed category: text to image Text 2 image pipeline category: Python API Python API for GenAI labels Nov 4, 2024
@eaidova
Copy link
Collaborator

eaidova commented Nov 4, 2024

@sbalandi could you please also add test run with --genai option in https://github.com/openvinotoolkit/openvino.genai/blob/master/.github/workflows/llm_bench-python.yml#L73

@github-actions github-actions bot added the category: GHA CI based on Github actions label Nov 4, 2024
@sbalandi
Copy link
Contributor Author

sbalandi commented Nov 4, 2024 •
edited
Loading

dreamlike-art/dreamlike-anime-1.0 is added @eaidova could you please confirm if it's okay, or too big ?
segmind/tiny-sd has unsupported type of scheduler DPMSolverMultistepScheduler

I have also tried segmind/tiny-sd-mxfinetune , this model has supported type of scheduler, but it fails in scheduler.cpp on creating pipeline, because scheduler.json has value -Infinity . @ilya-lavrenov is it expected ? should we handle it ?

@sbalandi sbalandi force-pushed the image_gen branch 2 times, most recently from 631d228 to 7f894f0 Compare November 4, 2024 18:41
@ilya-lavrenov
Copy link
Contributor

ilya-lavrenov commented Nov 4, 2024 •
edited
Loading

dreamlike-art/dreamlike-anime-1.0 is added @eaidova could you please confirm if it's okay, or too big ? segmind/tiny-sd has unsupported type of scheduler DPMSolverMultistepScheduler

I have also tried segmind/tiny-sd-mxfinetune , this model has supported type of scheduler, but it fails in scheduler.cpp on creating pipeline, because scheduler.json has value -Infinity . @ilya-lavrenov is it expected ? should we handle it ?

According to https://jsonlint.com/, it's invalid XML and I don't want to create WAs on GenAI for such cases (or only for popular models)

unsupported type of scheduler DPMSolverMultistepScheduler

You can override scheduler to support one via pipe.set_scheduler method.

@sbalandi
Copy link
Contributor Author

sbalandi commented Nov 4, 2024

dreamlike-art/dreamlike-anime-1.0 is added @eaidova could you please confirm if it's okay, or too big ? segmind/tiny-sd has unsupported type of scheduler DPMSolverMultistepScheduler
I have also tried segmind/tiny-sd-mxfinetune , this model has supported type of scheduler, but it fails in scheduler.cpp on creating pipeline, because scheduler.json has value -Infinity . @ilya-lavrenov is it expected ? should we handle it ?

According to https://jsonlint.com/, it's invalid XML and I don't want to create WAs on GenAI for such cases (or only for popular models)

unsupported type of scheduler DPMSolverMultistepScheduler

You can override scheduler to support one via pipe.set_scheduler method.

okay, thank you

@eaidova
Copy link
Collaborator

eaidova commented Nov 5, 2024

dreamlike-art/dreamlike-anime-1.0 is added @eaidova could you please confirm if it's okay, or too big ? segmind/tiny-sd has unsupported type of scheduler DPMSolverMultistepScheduler

I have also tried segmind/tiny-sd-mxfinetune , this model has supported type of scheduler, but it fails in scheduler.cpp on creating pipeline, because scheduler.json has value -Infinity . @ilya-lavrenov is it expected ? should we handle it ?

@sbalandi for me, any variant of model is ok, until it works in GHA

@ilya-lavrenov ilya-lavrenov removed the category: sampling Sampling / Decoding algorithms label Nov 5, 2024
@github-actions github-actions bot added the category: sampling Sampling / Decoding algorithms label Nov 5, 2024
@eaidova eaidova removed the category: sampling Sampling / Decoding algorithms label Nov 5, 2024
@eaidova eaidova added this pull request to the merge queue Nov 5, 2024
Merged via the queue into openvinotoolkit:master with commit 10b31e2 Nov 5, 2024
51 checks passed
@ilya-lavrenov
Copy link
Contributor

port to 2024.5 ?

@eaidova
Copy link
Collaborator

eaidova commented Nov 6, 2024 •
edited
Loading

port to 2024.5 ?

these changes present in release
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@popovaan popovaan popovaan left review comments

@eaidova eaidova eaidova approved these changes

@ilya-lavrenov ilya-lavrenov ilya-lavrenov left review comments

Assignees

@eaidova eaidova

Labels
category: GHA CI based on Github actions category: llm_bench Label for tool/llm_bench folder
Projects
None yet
Milestone
2024.5
Development

Successfully merging this pull request may close these issues.
4 participants
@sbalandi @eaidova @ilya-lavrenov @popovaan