Custom pipelines are broken

[nickkolok]

Describe the bug

It worked a couple of days ago, but today it stopped to. I wish I have never restarted none my space. Downgrading to diffusers 0.11.0, 0.12.0, 0.14.0 etc didn't help.

Please help me!

Reproduction

from diffusers import DiffusionPipeline, EulerAncestralDiscreteScheduler
import gradio as gr
import numpy as np
import imageio

from PIL import Image, ImageFile
ImageFile.LOAD_TRUNCATED_IMAGES = True
import glob


import torch
from time import gmtime, strftime


device = "cuda" if torch.cuda.is_available() else "cpu"
pipe = DiffusionPipeline.from_pretrained("NickKolok/meryl-stryfe-20230123-2300-6k-2400-steps_1", custom_pipeline="lpw_stable_diffusion", torch_dtype=torch.float32)
pipe.scheduler = EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config)
pipe.to(device)

def resize(height,img):
  baseheight = height
  img = Image.open(img)
  hpercent = (baseheight/float(img.size[1]))
  wsize = int((float(img.size[0])*float(hpercent)))
  img = img.resize((wsize,baseheight), Image.Resampling.LANCZOS)
  return img

def predict(prompt, negative_prompt):
    image = pipe(prompt=prompt, negative_prompt=negative_prompt, num_inference_steps=30, width=512, height=576).images[0]
    timestamp = strftime("%Y-%m-%d---%H-%M-%S", gmtime())
    #img = Image.open(image)
    image.save("./diffused__"+timestamp+"_.png")
    return glob.glob('./diffused_*.png')[-4:]

title="test"
description="test"

gr.Interface(
    fn=predict,
    inputs=[
        #gr.Image(source="upload", type="numpy", tool="sketch", elem_id="source_container"),
        gr.Textbox(label='What you want the AI to Generate, 77 Token limit'),
        gr.Textbox(label='What you Do Not want the AI to generate')
    ],
    outputs=[
        gr.Gallery(
            label="Generated images", show_label=False, elem_id="output_container_0"
        ).style(grid=[2], height="auto"),
        #gr.Image(elem_id="output_container_0"),
        #gr.Image(elem_id="output_container_1"),
    ],
    title=title,
    description=description,
    article = ""
).launch(max_threads=True, debug=True)

Logs

Downloading (…)_stable_diffusion.py:   0%|          | 0.00/13.9k [00:00<?, ?B/s]
Downloading (…)_stable_diffusion.py: 76.5kB [00:00, 17.6MB/s]                   
Traceback (most recent call last):
  File "app.py", line 16, in <module>
    pipe = DiffusionPipeline.from_pretrained("NickKolok/meryl-stryfe-20230123-2300-6k-2400-steps_1", custom_pipeline="lpw_stable_diffusion", torch_dtype=torch.float32)
  File "/home/user/.local/lib/python3.8/site-packages/diffusers/pipelines/pipeline_utils.py", line 655, in from_pretrained
    pipeline_class = get_class_from_dynamic_module(
  File "/home/user/.local/lib/python3.8/site-packages/diffusers/utils/dynamic_modules_utils.py", line 456, in get_class_from_dynamic_module
    return get_class_in_module(class_name, final_module.replace(".py", ""))
  File "/home/user/.local/lib/python3.8/site-packages/diffusers/utils/dynamic_modules_utils.py", line 163, in get_class_in_module
    module = importlib.import_module(module_path)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/user/.cache/huggingface/modules/diffusers_modules/git/lpw_stable_diffusion.py", line 13, in <module>
    from diffusers.image_processor import VaeImageProcessor
ModuleNotFoundError: No module named 'diffusers.image_processor'

System Info

huggingface spaces (It runs on cpu, on a free HF instance - if that helps.)
different diffusers versions (from 0.11.0 to 0.14.0 and even newest)

[nickkolok]

If I remove the part custom_pipeline="lpw_stable_diffusion", everything works (but not in the way I want). Can the issue be fixed by me - e.g. by adding any tweaks, hacks, etc?

[ghunkins]

Hi, having this error as well, even with pinned dependencies. What is going on?

[thundergolfer]

This is a more minimal reproduction:

import glob
from time import gmtime, strftime

import torch
from diffusers import DiffusionPipeline, EulerAncestralDiscreteScheduler


device = "cuda" if torch.cuda.is_available() else "cpu"
pipe = DiffusionPipeline.from_pretrained("NickKolok/meryl-stryfe-20230123-2300-6k-2400-steps_1", custom_pipeline="lpw_stable_diffusion", torch_dtype=torch.float32)
pipe.scheduler = EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config)
pipe.to(device)


if __name__ == "__main__":
    prompt="dummy"
    negative_prompt="negative dummy"
    image = pipe(prompt=prompt, negative_prompt=negative_prompt, num_inference_steps=30, width=512, height=576).images[0]
    timestamp = strftime("%Y-%m-%d---%H-%M-%S", gmtime())
    image.save("./diffused__"+timestamp+"_.png")
    print(glob.glob('./diffused_*.png')[-4:])

Error:

ImportError: cannot import name 'FromCkptMixin' from 'diffusers.loaders'
(/home/ubuntu/modal/venv/lib/python3.9/site-packages/diffusers/loaders.py)

This reproduction was run with:

diffusers==0.15.1
accelerate==0.18.0
torch==2.0.0+cu117

This import error is appearing in code that has not been redeployed, and is running the same container image that worked previously.

[thundergolfer]

9965cb5, merged 2 hours ago, is likely related.

[adhikjoshi]

same issue, @patrickvonplaten @SkyTNT @patil-suraj

why is diffusers custom_pipeline auto updating in background? without updating it? There was commit on lpw custom pipeline and it's code auto updated on remote production servers? This is vulnerability which needs to be fixed.

cannot import name 'FromCkptMixin' from 'diffusers.loaders' (/workspace/src/diffusers/src/diffusers/loaders.py)

[nickkolok]

Any idea how to monkey-patch it? Maybe we can wget a certain file from github using the appropriate commit hash?

[adhikjoshi]

PS: Avoid "lpw_stable_diffusion" community pipeline for better stability,

Previous reported same issue: #1607

How to migrate from "lpw_stable_diffusion"

pip install compel

from compel import Compel

from diffusers import StableDiffusionPipeline, UniPCMultistepScheduler

pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4")

pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)

prompt = "a red cat playing with a ((ball))"

generator = torch.Generator(device="cpu").manual_seed(33)

compel_proc = Compel(tokenizer=pipe.tokenizer, text_encoder=pipe.text_encoder)

prompt_embeds = compel_proc(prompt)

images = pipe(prompt_embeds=prompt_embeds, generator=generator, num_inference_steps=20).images[0]

[adhikjoshi]

Any idea how to monkey-patch it? Maybe we can wget a certain file from github using the appropriate commit hash?

you can't:

#1607

[ghunkins]

You can pin the dependency for dynamic imports. We use 0.15.1.

self.pipe = diffusers.DiffusionPipeline.from_pretrained(
                self.CONFIG["base"]["model_cache"],
                custom_pipeline="lpw_stable_diffusion",
                revision='0.15.1',
                custom_revision='0.15.1',
                torch_dtype=dtype,
            ).to("cuda")

[ghunkins]

Compel uses different syntax, so it's difficult to migrate existing workflows.

[adhikjoshi]

Compel uses different syntax, so it's difficult to migrate existing workflows.

True, (()) is not there. Thinking of taking prompt weighting of lpw as separate lib, to get out of this dependency.

[ghunkins]

This would be incredible. Or having an option to switch syntaxes via compel to unify it all.

[nickkolok]

custom_revision='0.15.1'

At least, it should be emphasized in the docs. That's really non-obvious.

@ghunkins, thank you very much!

[patrickvonplaten]

Didn't know that so many people use the community pipeline!

For now can you try to always pass custom_revision?

Also a better maintained version might also be the compel library maybe?