Rework LoRA section, fix incongruent package versions required by different samples #567

yatarkan · 2024-07-01T14:45:18Z

No description provided.

Wovchena · 2024-07-03T13:52:44Z

image_generation/stable_diffusion_1_5/cpp/README.md

@@ -32,7 +32,7 @@ conda env config vars set LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH
 git submodule update --init
 # Reactivate Conda environment after installing dependencies and setting env vars
 conda activate openvino_sd_cpp
-python -m pip install -r requirements.txt
+python -m pip install -r ../../requirements.txt
 python -m pip install ../../../thirdparty/openvino_tokenizers/[transformers]
 ```
 2. Download a huggingface SD v1.5 model like:


optimum-cli export openvino below is in [dreamlike-anime-1.0](https://huggingface.co/dreamlike-art/dreamlike-anime-1.0) section even though it they aren't related. Please rework this. I think you can move down as an example of

There are various LoRA models on https://civitai.com/tag/lora

[dreamlike-anime-1.0](https://huggingface.co/dreamlike-art/dreamlike-anime-1.0) section because it's described later.

runwayml/stable-diffusion-v1-5 and dreamlike-art/dreamlike-anime-1.0 are Stable Diffusion models exported with optimum-cli, not the LoRA models. I suppose it is might be useful to show the user which example SD models he can download and export.
Reworked this section and made the messaging more clear

Wovchena · 2024-07-03T13:53:20Z

image_generation/stable_diffusion_1_5/cpp/README.md


-Refer to [python pipeline blog](https://blog.openvino.ai/blog-posts/enable-lora-weights-with-stable-diffusion-controlnet-pipeline).
-The safetensor model is loaded via [safetensors.h](https://github.com/hsnyder/safetensors.h). The layer name and weight are modified with `Eigen` library and inserted into the SD models with `ov::pass::MatcherPass` in the file [common/diffusers/src/lora.cpp](https://github.com/openvinotoolkit/openvino.genai/blob/master/image_generation/common/diffusers/src/lora.cpp).
+Low-Rank Adaptation(LoRA) is a novel technique introduced to deal with the problem of fine-tuning Diffusers and Large Language Models (LLMs). In the case of Stable Diffusion fine-tuning, LoRA can be applied to the cross-attention layers for the image representations with the latent described.


Suggested change

Low-Rank Adaptation(LoRA) is a novel technique introduced to deal with the problem of fine-tuning Diffusers and Large Language Models (LLMs). In the case of Stable Diffusion fine-tuning, LoRA can be applied to the cross-attention layers for the image representations with the latent described.

Low-Rank Adaptation(LoRA) is technique introduced to deal with the problem of fine-tuning Diffusers and Large Language Models (LLMs). In the case of Stable Diffusion fine-tuning, LoRA can be applied to the cross-attention layers for the image representations with the latent described.

it won't be novel after some time. Or maybe it's already not novel :)

Wovchena · 2024-07-03T13:57:05Z

image_generation/stable_diffusion_1_5/cpp/README.md

+
+SD model [dreamlike-anime-1.0](https://huggingface.co/dreamlike-art/dreamlike-anime-1.0) and LoRA [soulcard model](https://civitai.com/models/67927?modelVersionId=72591) are tested in this pipeline.
+
+There are various LoRA models on https://civitai.com/tag/lora and on HuggingFace, you can consider to choose your own LoRA model in safetensor format. 


Move that line to the LoRA Weights Format section. It's a natural question to ask to see examples of the weights with the described format. That would leave Using LoRA Weights section with just two sentences. And

SD model dreamlike-anime-1.0 and LoRA soulcard model are tested in this pipeline.

isn't informative. I suggest merging them and naming the merged section LoRA Weights

Fixed, sections are merged

Wovchena · 2024-07-03T14:01:03Z

image_generation/requirements.txt

@@ -0,0 +1,5 @@
+--extra-index-url https://download.pytorch.org/whl/cpu
+torch==2.2.2+cpu; sys_platform != 'darwin'
+torch==2.2.2; sys_platform == 'darwin'


Please, add a comment explaining the diff

Wovchena · 2024-07-03T14:02:58Z

image_generation/stable_diffusion_1_5/cpp/README.md


 * Generate image without lora `./build/stable_diffusion -r`

   ![](./without_lora.bmp)

-* Generate image with soulcard lora `./build/stable_diffusion -r`
+* Generate image with soulcard lora `./build/stable_diffusion -r -l path/to/soulcard.safetensors`

   ![](./soulcard_lora.bmp)



While you are nearby, please replace is differ to differ below.

Fixed for both SD and LCM

Wovchena · 2024-07-03T14:04:57Z

samples/requirements.txt

@@ -2,3 +2,4 @@
 optimum[openvino]==1.20.0
 einops==0.8.0  # For Qwen
 transformers_stream_generator==0.0.5  # For Qwen
+-r ../image_generation/requirements.txt


Reminder to inverse the dependency or introduce a common requirements.txt

image_generation/stable_diffusion_1_5/cpp/README.md

Wovchena · 2024-07-08T07:47:05Z

image_generation/requirements.txt

+torch==2.2.2+cpu; sys_platform != 'darwin'
+torch==2.2.2; sys_platform == 'darwin'  # Torch wheel for 2.2.2+cpu does not exist for macOS, using 2.2.2 instead
+diffusers==0.27.2
+optimum-intel[openvino]==1.18.0


Suggested change

optimum-intel[openvino]==1.18.0

samples already have optimum[openvino]==1.20.0. That's strange that pip didn't detect a conflict

From samples optimum-intel is installed with version >=1.16.0, so there is no conflict.
But I removed duplicated dependencies from image generation

Wovchena · 2024-07-08T07:47:45Z

image_generation/requirements.txt

@@ -0,0 +1,6 @@
+--extra-index-url https://download.pytorch.org/whl/cpu
+torch==2.2.2+cpu; sys_platform != 'darwin'
+torch==2.2.2; sys_platform == 'darwin'  # Torch wheel for 2.2.2+cpu does not exist for macOS, using 2.2.2 instead


I just realized. I'd expect torch to come with optimum. Can torch be removed? Maybe the same is applicable to diffusers

Duplicated dependencies removed keeping only diffusers in requirements

…ferent samples (#567)

yatarkan added 10 commits July 1, 2024 18:44

Remove transformers and hf hub dependencies from SD dependencies

3bb2808

Try using samples requirements for SD workflow

edf4649

Merge branch 'master' into yt/update-lora-section

d7778dc

Move common requirements for SD and LCM to image_generation dir

63d9cc4

Add LCM link to image generation readme

82c47ab

Align optimum intel version

175f368

Add extra index url to install command for SD and LCM

c9d52cc

Fix optimum intel version

3d9c36d

Include image generation requirements to samples requirements

9df7c54

Rework LoRA section in SD

e6729b3

yatarkan changed the title ~~Fix incongruent package versions required by different samples~~ Rework LoRA section, fix incongruent package versions required by different samples Jul 2, 2024

yatarkan requested a review from Wovchena July 2, 2024 16:50

Add platform specific requirements for image generation

c1f74dd

Wovchena requested changes Jul 3, 2024

View reviewed changes

Wovchena reviewed Jul 3, 2024

View reviewed changes

image_generation/stable_diffusion_1_5/cpp/README.md Outdated Show resolved Hide resolved

yatarkan added 4 commits July 5, 2024 12:06

Include samples requirements in image generation

07c0f0f

Rework models and lora sections

5988f3e

Add updated lora section to LCM readme

efbf620

Merge branch 'master' into yt/update-lora-section

79e7789

yatarkan requested a review from Wovchena July 5, 2024 09:38

Wovchena approved these changes Jul 8, 2024

View reviewed changes

Wovchena marked this pull request as ready for review July 8, 2024 07:44

Wovchena enabled auto-merge July 8, 2024 07:44

Wovchena disabled auto-merge July 8, 2024 07:45

Wovchena requested changes Jul 8, 2024

View reviewed changes

yatarkan added 4 commits July 8, 2024 13:48

Merge branch 'master' into yt/update-lora-section

297d58a

Test skip optimum-intel and torch deps from image generation

59bff2b

Remove all deps from image generation

942458f

Add diffusers dependency to image generation

7705af5

yatarkan requested a review from Wovchena July 8, 2024 11:42

Wovchena approved these changes Jul 8, 2024

View reviewed changes

Wovchena added this pull request to the merge queue Jul 8, 2024

github-merge-queue bot pushed a commit that referenced this pull request Jul 8, 2024

Rework LoRA section, fix incongruent package versions required by dif…

61953d0

…ferent samples (#567)

Merged via the queue into openvinotoolkit:master with commit a21ffbd Jul 8, 2024
15 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rework LoRA section, fix incongruent package versions required by different samples #567

Rework LoRA section, fix incongruent package versions required by different samples #567

yatarkan commented Jul 1, 2024

Wovchena Jul 3, 2024

yatarkan Jul 5, 2024 •

edited

Loading

Wovchena Jul 3, 2024

yatarkan Jul 5, 2024

Wovchena Jul 3, 2024

yatarkan Jul 5, 2024

Wovchena Jul 3, 2024

yatarkan Jul 5, 2024

Wovchena Jul 3, 2024

yatarkan Jul 5, 2024

Wovchena Jul 3, 2024

yatarkan Jul 5, 2024

Wovchena Jul 8, 2024

yatarkan Jul 8, 2024

Wovchena Jul 8, 2024 •

edited

Loading

yatarkan Jul 8, 2024

	Low-Rank Adaptation(LoRA) is a novel technique introduced to deal with the problem of fine-tuning Diffusers and Large Language Models (LLMs). In the case of Stable Diffusion fine-tuning, LoRA can be applied to the cross-attention layers for the image representations with the latent described.
	Low-Rank Adaptation(LoRA) is technique introduced to deal with the problem of fine-tuning Diffusers and Large Language Models (LLMs). In the case of Stable Diffusion fine-tuning, LoRA can be applied to the cross-attention layers for the image representations with the latent described.


		SD model [dreamlike-anime-1.0](https://huggingface.co/dreamlike-art/dreamlike-anime-1.0) and LoRA [soulcard model](https://civitai.com/models/67927?modelVersionId=72591) are tested in this pipeline.

		There are various LoRA models on https://civitai.com/tag/lora and on HuggingFace, you can consider to choose your own LoRA model in safetensor format.

Rework LoRA section, fix incongruent package versions required by different samples #567

Rework LoRA section, fix incongruent package versions required by different samples #567

Conversation

yatarkan commented Jul 1, 2024

Choose a reason for hiding this comment

yatarkan Jul 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Wovchena Jul 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yatarkan Jul 5, 2024 •

edited

Loading

Wovchena Jul 8, 2024 •

edited

Loading