lpw_stable_diffusion pipeline is broken. Again.

[nickkolok]

Describe the bug

Just as in #3199 , lpw_stable_diffusion pipeline has suddenly broke today on an ordinary Factory reboot of my space.

Reproduction

I've created a public HF space to reproduce the bug:
https://huggingface.co/spaces/NickKolok/test-20230424-bug-reprod/

Feel free to duplicate it and play with it. If you remove the custom pipeline, everything runs.

Logs

/home/user/.local/lib/python3.8/site-packages/huggingface_hub/file_download.py:645: FutureWarning: `cached_download` is the legacy way to download files from the HF hub, please consider upgrading to `hf_hub_download`
  warnings.warn(

Downloading (…)_stable_diffusion.py:   0%|          | 0.00/10.5k [00:00<?, ?B/s]Could not locate the pipeline.py inside lpw_stable_diffusion.
Traceback (most recent call last):
  File "app.py", line 16, in <module>
    pipe = DiffusionPipeline.from_pretrained(
  File "/home/user/.local/lib/python3.8/site-packages/diffusers/pipelines/pipeline_utils.py", line 856, in from_pretrained
    cached_folder = cls.download(
  File "/home/user/.local/lib/python3.8/site-packages/diffusers/pipelines/pipeline_utils.py", line 1227, in download
    pipeline_class = _get_pipeline_class(
  File "/home/user/.local/lib/python3.8/site-packages/diffusers/pipelines/pipeline_utils.py", line 335, in _get_pipeline_class
    return get_class_from_dynamic_module(
  File "/home/user/.local/lib/python3.8/site-packages/diffusers/utils/dynamic_modules_utils.py", line 445, in get_class_from_dynamic_module
    final_module = get_cached_module_file(
  File "/home/user/.local/lib/python3.8/site-packages/diffusers/utils/dynamic_modules_utils.py", line 285, in get_cached_module_file
    resolved_module_file = cached_download(
  File "/home/user/.local/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py", line 120, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/user/.local/lib/python3.8/site-packages/huggingface_hub/file_download.py", line 788, in cached_download
    http_get(
  File "/home/user/.local/lib/python3.8/site-packages/huggingface_hub/file_download.py", line 543, in http_get
    raise EnvironmentError(
OSError: Consistency check failed: file should be of size 10495 but has size 57300 ((…)_stable_diffusion.py).
We are sorry for the inconvenience. Please retry download and pass `force_download=True, resume_download=False` as argument.
If the issue persists, please let us know by opening an issue on https://github.com/huggingface/huggingface_hub.

Downloading (…)_stable_diffusion.py: 57.3kB [00:00, 491kB/s]

System Info

HuggingFace free space

[patrickvonplaten]

Hey @nickkolok, as stated in the issue, can you try force_download=True?

[patrickvonplaten]

Also see: #3171 (comment)

[nickkolok]

Hey @nickkolok, as stated in the issue, can you try force_download=True?

The error remain the same (see the space above).

pipe = DiffusionPipeline.from_pretrained(
    "admruul/anything-v3.0",
    #Comment the following line to make it launch
    custom_pipeline="lpw_stable_diffusion",
    custom_revision='0.15.1',
    force_download=True,
    resume_download=False,
    torch_dtype=torch.float32
)

Also, there is a slight difference between "we don't actively maintain the pipeline, so in future releases it can be unstable" and "the pipeline is broken without any code change and without any direct reasons", isn't it?

[nickkolok]

Seems to work on diffusers==0.12.0

[mack-w]

Can confirm on 0.15.1, worked yesterday, broken today (on a different computer).
Tried force_download and resume_download, no good.

[Wauplin]

The issues lies in the latest release of huggingface_hub (0.14.0) from yesterday. We are working on a fix. In the meantime you can downgrade huggingface_hub to 0.13.4 and it should continue to work.

I'll keep you posted about a fix.

[mack-w]

The issues lies in the latest release of huggingface_hub (0.14.0) from yesterday. We are working on a fix. In the meantime you can downgrade huggingface_hub to 0.13.4 and it should continue to work.

I'll keep you posted about a fix.

Thank you for the workaround. It is working again by now. @nickkolok

[Wauplin]

@nickkolok @mack-w Issue has been fixed and a patch release (huggingface_hub==0.14.1) has been published. This should close this issue :)

[yiyixuxu]

@nickkolok does it work for you now?
will close this issue once you confirm it's working now 🤗

[nickkolok]

Unfortunately, I have to state that for me the problem persist. I have to stick to workaround (huggingface_hub==0.13.4) above for that I'm really grateful.

[yiyixuxu]

@nickkolok

the problem persist for huggingface_hub==0.14.1 or 0.14.0 ?
think you have to do pip install -U huggingface_hub to update

YiYi

[nickkolok]

I still can reproduce the bug, although it has become more subtle

pipe = DiffusionPipeline.from_pretrained(
    "admruul/anything-v3.0",
    #Comment the following line to make it launch
    custom_pipeline="lpw_stable_diffusion",
    #custom_revision='0.14.0',
    #force_download=True,
    #resume_download=False,
    torch_dtype=torch.float32
)

Here, huggingface_hub is listed amid the requirements ( https://huggingface.co/spaces/NickKolok/test-20230424-bug-reprod/blob/main/requirements.txt ), but it's version is not specified. If I specify huggingface_hub==0.14.1, everything fixes up.

Hope it helps.

[Wauplin]

Here, huggingface_hub is listed amid the requirements but it's version is not specified. If I specify huggingface_hub==0.14.1, everything fixes up.

I think this is because the Space has cached a Docker image with an old version of huggingface_hub (I guess 0.14.0). If you go on the settings page of the Space and click on "Factory reboot" it will recreate entirely the docker image and reinstall with the latest dependency.

Another solution is just to add huggingface_hub>=0.14.1 in your requirements.txt file to force it.

[nickkolok]

I've done a Factory reboot, but the issue is still here...
https://huggingface.co/spaces/NickKolok/test-20230424-bug-reprod

[Wauplin]

@nickkolok That's really weird 😕
I literally just duplicated your Space without changing any code and it works just fine: https://huggingface.co/spaces/Wauplin/test-20230424-bug-reprod-duplicate

This has to do with a dependency version issue. I can't explain why the factory reboot did not solve it for you but if you add huggingface_hub>=0.14.1 in your requirements.txt file, it should work.

[nickkolok]

Thank you all for pointing out the workaround! For me it works like a charm. If anyone from the amazing HuggingFace team wants to dive deeper into the bug with factory reboot, I will be really glad to transfer the affected space to them.

Otherwise, the issue can be surely closed.