From acaaec2517bece77650362c89af3ab694ca4d078 Mon Sep 17 00:00:00 2001 From: Chengsong Zhang Date: Mon, 4 Sep 2023 15:12:40 -0500 Subject: [PATCH 1/4] prepare for v1.3.0 update --- .gitignore | 3 ++- README.md | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/.gitignore b/.gitignore index a1db249e..429a954b 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,3 @@ __pycache__ -model/*.ckpt \ No newline at end of file +model/*.ckpt +model/*.pth \ No newline at end of file diff --git a/README.md b/README.md index 1c0640ac..96f33f52 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,8 @@ You might also be interested in another extension I created: [Segment Anything f - `2023/07/20` [v1.1.0](https://github.com/continue-revolution/sd-webui-animatediff/releases/tag/v1.1.0): fix gif duration, add loop number, remove auto-download, remove xformers, remove instructions on gradio UI, refactor README, add [sponsor](#sponsor) QR code. - `2023/07/24` [v1.2.0](https://github.com/continue-revolution/sd-webui-animatediff/releases/tag/v1.2.0): fix incorrect insertion of motion modules, add option to change path to save motion modules in Settings/AnimateDiff, fix loading different motion modules. -- `2023/07/27` [v1.2.1](https://github.com/continue-revolution/sd-webui-animatediff/releases/tag/v1.2.1): add hash calculation of motion modules (you can disable it in `Settings/AnimateDiff`) +- `2023/07/27` [v1.2.1](https://github.com/continue-revolution/sd-webui-animatediff/releases/tag/v1.2.1): add hash calculation of motion modules (you can disable it in `Settings/AnimateDiff`) +- `2023/09/04` [v1.3.0](https://github.com/continue-revolution/sd-webui-animatediff/releases/tag/v1.3.0): support any community models with the same architecture, improve quality via [#63](https://github.com/continue-revolution/sd-webui-animatediff/issues/63) (credit to [@TDS4874](https://github.com/TDS4874)) ## TODO This TODO list will most likely be resolved sequentially. From bcea44539dd899ec47499a56deb1ea075440b82c Mon Sep 17 00:00:00 2001 From: Chengsong Zhang Date: Mon, 4 Sep 2023 15:32:36 -0500 Subject: [PATCH 2/4] readme model zoo --- README.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 96f33f52..a4b0a912 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ You might also be interested in another extension I created: [Segment Anything f ## How to Use 1. Install this extension via link. -2. Download motion modules from [Google Drive](https://drive.google.com/drive/folders/1EqLC65eR1-W-sGD0Im7fkED6c8GkiNFI) | [HuggingFace](https://huggingface.co/guoyww/animatediff) | [CivitAI](https://civitai.com/models/108836) | [Baidu NetDisk](https://pan.baidu.com/s/18ZpcSM6poBqxWNHtnyMcxg?pwd=et8y). You only need to download one of `mm_sd_v14.ckpt` | `mm_sd_v15.ckpt`. Put the model weights under `sd-webui-animatediff/model/`. DO NOT change model filename. +2. Download motion modules and put the model weights under `sd-webui-animatediff/model/`. DO NOT change model filename. See [model zoo](#motion-module-model-zoo) for a list of available motion modules. 3. Go to txt2img if you want to try txt2gif and img2img if you want to try img2gif. 4. Choose an SD1.5 checkpoint, write prompts, set configurations such as image width/height. If you want to generate multiple GIFs at once, please change batch number, instead of batch size. 5. Enable AnimateDiff extension, and set up each parameter, and click `Generate`. @@ -22,6 +22,10 @@ You might also be interested in another extension I created: [Segment Anything f 1. **Loop number** — How many times the GIF is played. A value of `0` means the GIF never stops playing. 5. You should see the output GIF on the output gallery. You can access GIF output at `stable-diffusion-webui/outputs/{txt2img or img2img}-images/AnimateDiff`. You can also access image frames at `stable-diffusion-webui/outputs/{txt2img or img2img}-images/{date}`. +## Motion Module Model Zoo +- `mm_sd_v14.ckpt` & `mm_sd_v15.ckpt` by [@guoyww](https://github.com/guoyww): [Google Drive](https://drive.google.com/drive/folders/1EqLC65eR1-W-sGD0Im7fkED6c8GkiNFI) | [HuggingFace](https://huggingface.co/guoyww/animatediff) | [CivitAI](https://civitai.com/models/108836) | [Baidu NetDisk](https://pan.baidu.com/s/18ZpcSM6poBqxWNHtnyMcxg?pwd=et8y) +- `mm-Stabilized_high.pth` & `mm-Stabbilized_mid.pth` by [@manshoety](https://huggingface.co/manshoety): [HuggingFace](https://huggingface.co/manshoety/AD_Stabilized_Motion/tree/main) + ## Update - `2023/07/20` [v1.1.0](https://github.com/continue-revolution/sd-webui-animatediff/releases/tag/v1.1.0): fix gif duration, add loop number, remove auto-download, remove xformers, remove instructions on gradio UI, refactor README, add [sponsor](#sponsor) QR code. @@ -31,7 +35,6 @@ You might also be interested in another extension I created: [Segment Anything f ## TODO This TODO list will most likely be resolved sequentially. -- [ ] greyer sample - [ ] other attention optimization (e.g. sdp) - [ ] img2img - [ ] [token](https://github.com/continue-revolution/sd-webui-animatediff/issues/4) From 8f9bedbd544f49302d20133351c9f238e5ecbb9b Mon Sep 17 00:00:00 2001 From: Chengsong Zhang Date: Mon, 4 Sep 2023 16:17:28 -0500 Subject: [PATCH 3/4] any community model supported --- README.md | 2 +- scripts/animatediff.py | 55 +++++++++++++++++++++++++++++++++--------- 2 files changed, 44 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index a4b0a912..3cd86763 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ You might also be interested in another extension I created: [Segment Anything f ## How to Use 1. Install this extension via link. -2. Download motion modules and put the model weights under `sd-webui-animatediff/model/`. DO NOT change model filename. See [model zoo](#motion-module-model-zoo) for a list of available motion modules. +2. Download motion modules and put the model weights under `sd-webui-animatediff/model/`. See [model zoo](#motion-module-model-zoo) for a list of available motion modules. 3. Go to txt2img if you want to try txt2gif and img2img if you want to try img2gif. 4. Choose an SD1.5 checkpoint, write prompts, set configurations such as image width/height. If you want to generate multiple GIFs at once, please change batch number, instead of batch size. 5. Enable AnimateDiff extension, and set up each parameter, and click `Generate`. diff --git a/scripts/animatediff.py b/scripts/animatediff.py index 9696b5a2..b3403f13 100644 --- a/scripts/animatediff.py +++ b/scripts/animatediff.py @@ -29,6 +29,16 @@ def mm_tes_forward(self, x, emb, context=None): groupnorm32_original_forward = GroupNorm32.forward +class ToolButton(gr.Button, gr.components.FormComponent): + """Small button with single emoji as text, fits inside gradio forms""" + + def __init__(self, **kwargs): + super().__init__(variant="tool", **kwargs) + + def get_block_name(self): + return "button" + + class AnimateDiffScript(scripts.Script): motion_module: MotionWrapper = None @@ -56,8 +66,25 @@ def remove_motion_module(self): gc.collect() def ui(self, is_img2img): + model_dir = shared.opts.data.get("animatediff_model_path", os.path.join(script_dir, "model")) + if not os.path.isdir(model_dir): + os.mkdir(model_dir) + model_list = [f for f in os.listdir(model_dir) if f != ".gitkeep"] with gr.Accordion('AnimateDiff', open=False): - model = gr.Dropdown(choices=["mm_sd_v15.ckpt", "mm_sd_v14.ckpt"], value="mm_sd_v15.ckpt", label="Motion module", type="value") + with gr.Row(): + def refresh_models(*inputs): + new_model_list = [f for f in os.listdir(model_dir) if f != ".gitkeep"] + dd = inputs[0] + if dd in new_model_list: + selected = dd + elif len(new_model_list) > 0: + selected = new_model_list[0] + else: + selected = None + return gr.Dropdown.update(choices=new_model_list, value=selected) + model = gr.Dropdown(choices=model_list, value=model_list[0], label="Motion module", type="value") + refresh_model = ToolButton(value='\U0001f504') + refresh_model.click(refresh_models, model, model) with gr.Row(): enable = gr.Checkbox(value=False, label='Enable AnimateDiff') video_length = gr.Slider(minimum=1, maximum=24, value=16, step=1, label="Number of frames", precision=0) @@ -75,16 +102,21 @@ def inject_motion_modules(self, p: StableDiffusionProcessing, model_name="mm_sd_ if not os.path.isfile(model_path): raise RuntimeError("Please download models manually.") if AnimateDiffScript.motion_module is None or AnimateDiffScript.motion_module.mm_type != model_name: - if shared.opts.data.get("animatediff_check_hash", True): - def get_mm_hash(model_name="mm_sd_v15.ckpt"): - if model_name == "mm_sd_v14.ckpt": - return 'aa7fd8a200a89031edd84487e2a757c5315460eca528fa70d4b3885c399bffd5' - elif model_name == "mm_sd_v15.ckpt": - return 'cf16ea656cb16124990c8e2c70a29c793f9841f3a2223073fac8bd89ebd9b69a' - else: - raise RuntimeError(f"Unsupported model filename {model_name}. Should be one of mm_sd_v14.ckpt or mm_sd_v15.ckpt") - if hashes.sha256(model_path, f"AnimateDiff/{model_name}") != get_mm_hash(model_name): - raise RuntimeError(f"{model_name} hash mismatch. You probably need to re-download the motion module.") + def get_mm_hash(model_name="mm_sd_v15.ckpt"): + model_hash = hashes.sha256(model_path, f"AnimateDiff/{model_name}") + if model_hash == 'aa7fd8a200a89031edd84487e2a757c5315460eca528fa70d4b3885c399bffd5': + self.logger.info('You are using mm_sd_14.ckpt, which has been tested and supported.') + elif model_hash == "cf16ea656cb16124990c8e2c70a29c793f9841f3a2223073fac8bd89ebd9b69a": + self.logger.info('You are using mm_sd_15.ckpt, which has been tested and supported.') + elif model_hash == "0aaf157b9c51a0ae07cb5d9ea7c51299f07bddc6f52025e1f9bb81cd763631df": + self.logger.info('You are using mm-Stabilized_high.pth, which has been tested and supported.') + elif model_hash == '39de8b71b1c09f10f4602f5d585d82771a60d3cf282ba90215993e06afdfe875': + self.logger.info('You are using mm-Stabilized_mid.pth, which has been tested and supported.') + else: + self.logger.warn(f"Your model {model_name} has not been tested and supported. " + "Either your download is incomplete or your model has not been tested. " + "Please use at your own risk.") + get_mm_hash(model_name) self.logger.info(f"Loading motion module {model_name} from {model_path}") mm_state_dict = torch.load(model_path, map_location=device) AnimateDiffScript.motion_module = MotionWrapper(model_name) @@ -157,7 +189,6 @@ def postprocess(self, p: StableDiffusionProcessing, res: Processed, enable_anima def on_ui_settings(): section = ('animatediff', "AnimateDiff") shared.opts.add_option("animatediff_model_path", shared.OptionInfo(os.path.join(script_dir, "model"), "Path to save AnimateDiff motion modules", gr.Textbox, section=section)) - shared.opts.add_option("animatediff_check_hash", shared.OptionInfo(True, "Check hash for motion modules. Disable checking if you want to use your own.", gr.Checkbox, section=section)) script_callbacks.on_ui_settings(on_ui_settings) From 3242c548d6105037c39a0a086a39a056f43c0f79 Mon Sep 17 00:00:00 2001 From: Chengsong Zhang Date: Mon, 4 Sep 2023 19:34:12 -0500 Subject: [PATCH 4/4] readme --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 3cd86763..660e0368 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,7 @@ You might also be interested in another extension I created: [Segment Anything f - `2023/07/20` [v1.1.0](https://github.com/continue-revolution/sd-webui-animatediff/releases/tag/v1.1.0): fix gif duration, add loop number, remove auto-download, remove xformers, remove instructions on gradio UI, refactor README, add [sponsor](#sponsor) QR code. - `2023/07/24` [v1.2.0](https://github.com/continue-revolution/sd-webui-animatediff/releases/tag/v1.2.0): fix incorrect insertion of motion modules, add option to change path to save motion modules in Settings/AnimateDiff, fix loading different motion modules. -- `2023/07/27` [v1.2.1](https://github.com/continue-revolution/sd-webui-animatediff/releases/tag/v1.2.1): add hash calculation of motion modules (you can disable it in `Settings/AnimateDiff`) +- `2023/07/27` [v1.2.1](https://github.com/continue-revolution/sd-webui-animatediff/releases/tag/v1.2.1): ~~add hash calculation of motion modules (you can disable it in `Settings/AnimateDiff`).~~ Starting from v1.3.0, hash will always be calculated, but nothing will prevent you from using any motion modules you want. - `2023/09/04` [v1.3.0](https://github.com/continue-revolution/sd-webui-animatediff/releases/tag/v1.3.0): support any community models with the same architecture, improve quality via [#63](https://github.com/continue-revolution/sd-webui-animatediff/issues/63) (credit to [@TDS4874](https://github.com/TDS4874)) ## TODO