HelloMeme: Integrating Spatial Knitting Attentions to Embed High-Level and Fidelity-Rich Conditions in Diffusion Models

Shengkai Zhang, Nianhong Jiao, Tian Li, Chaojie Yang, Chenhui Xue^*, Boya Niu^*, Jun Gao

HelloVision | HelloGroup Inc.

^* Intern

🔆 New Features/Updates

• ☐ ExperimentsOnSKAttentions for ablation experiments.

• ☐ SDXL version.

• ✅ 02/09/2025 HelloMemeV3 is now available. YouTube Demo

• ✅ 12/17/2024 Added modelscope Demo.

• ✅ 12/13/2024 Rewrite the code for the Gradio app.

• ✅ 12/12/2024 Added HelloMeme V2 (synchronize code from the ComfyUI repo).

• ✅ 11/14/2024 Added the HMControlNet2 module

• ✅ 11/12/2024 Added a newly fine-tuned version of Animatediff with a patch size of 12, which uses less VRAM (Tested on 2080Ti).

• ✅ 11/5/2024 ComfyUI interface for HelloMeme.

• ✅ 11/1/2024 Release the code for the core functionalities..

Introduction

This repository contains the official code implementation of the paper HelloMeme. Any updates related to the code or models from the paper will be posted here. The code for the ablation experiments discussed in the paper will be added to the ExperimentsOnSKAttentions section. Additionally, we plan to release a ComfyUI interface for HelloMeme, with updates posted here as well.

Comfyui Installation

Keyword of ComfyUI Manager : hellomeme-api

Getting Started

1. Create a Conda Environment

conda create -n hellomeme python=3.10.11
conda activate hellomeme

2. Install PyTorch and FFmpeg

To install the latest version of PyTorch, please refer to the official PyTorch website for detailed installation instructions. Additionally, the code will invoke the system's ffmpeg command for video and audio editing, so the runtime environment must have ffmpeg pre-installed. For installation guidance, please refer to the official FFmpeg website.

3. Install dependencies

pip install -r requirements.txt

4. Clone the repository

git clone https://github.com/HelloVision/HelloMeme
cd HelloMeme

5. Run the code

python inference_image.py # for image generation
python inference_video.py # for video generation
python app.py # for Gradio App

After run the app, all models will be downloaded.

Examples

Image Generation

The input for the image generation script inference_image.py consists of a reference image and a drive image, as shown in the figure below:

Reference Image

Drive Image

The output of the image generation script is shown below:

Based on SD1.5

Based on disneyPixarCartoon

Video Generation

The input for the video generation script inference_video.py consists of a reference image and a drive video, as shown in the figure below:

Reference Image

Drive Video

The output of the video generation script is shown below:

Based on epicrealism

Based on disneyPixarCartoon

Acknowledgements

Thanks to 🤗 for providing diffusers, which has greatly enhanced development efficiency in diffusion-related work. We also drew considerable inspiration from MagicAnimate and EMO, and Animatediff allowed us to implement the video version at a very low cost. Finally, we thank our colleagues Shengjie Wu and Zemin An, whose foundational modules played a significant role in this work.

Citation

@misc{zhang2024hellomemeintegratingspatialknitting,
        title={HelloMeme: Integrating Spatial Knitting Attentions to Embed High-Level and Fidelity-Rich Conditions in Diffusion Models}, 
        author={Shengkai Zhang and Nianhong Jiao and Tian Li and Chaojie Yang and Chenhui Xue and Boya Niu and Jun Gao},
        year={2024},
        eprint={2410.22901},
        archivePrefix={arXiv},
        primaryClass={cs.CV},
        url={https://arxiv.org/abs/2410.22901}, 
  }

Contact

Shengkai Zhang (songkey@pku.edu.cn)