StreamDiffusionIO's pipeline design is based on StreamDiffusion, but especially allows using different text prompt on different samples in the denoising batch respectively but consistently.
A natural application of StreamDiffusionIO is to render text streams into image streams, as in Streaming Kanji.
- Streaming with LDM
- Streaming with LCM
conda create -n StreamDiffusionIO python=3.10
conda activate StreamDiffusionIOppip install StreamDiffusionIOgit clone https://github.com/AgainstEntropy/StreamDiffusionIO.git
pip install --editable ./StreamDiffusionIO/# For user
pip install StreamDiffusionIO[xformers]
# For dev
pip install -e '.[xformers]'Or install xformers manually:
pip install xformers --index-url https://download.pytorch.org/whl/cu124StreamDiffusionIO is very similar to StreamDiffusion, but even more lightweight. One can use the pipeline with only a few lines of codes.
import torch
from StreamDiffusionIO import LatentConsistencyModelStreamIO
device = "cuda" if torch.cuda.is_available() else "cpu"
model_id_or_path = "/path/to/stable-diffusion-v1-5"
lora_path = "/path/to/lora/pytorch_lora_weights.safetensors"
lcm_lora_path = "/path/to/lcm-lora/pytorch_lora_weights.safetensors"
stream = LatentConsistencyModelStreamIO(
model_id_or_path=model_id_or_path,
lcm_lora_path=lcm_lora_path,
lora_dict={lora_path: 1},
resolution=128,
device=device,
)
text = "Today I saw a beautiful sunset and it made me feel so happy."
prompt_list = text.split()
# to simulate a text stream
for prompt in prompt_list:
image, text = stream(prompt) # stream returns None during warmup
if image is not None:
print(text)
display(image)
# Continue to display the remaining images in the stream
while True:
image, text = stream(prompt)
print(text)
display(image)
if stream.stop():
breakNote the text returnded from the stream is the corresponding text prompt used to generating the returned image.
Please follow the Jupyter notebooks in examples to see details.