IPAdapter + Sliding window in animatediff is not working #904

Gutianpei · 2024-05-23T22:27:07Z

Describe the bug

The weights of ipadapter can be correctly applied! Thanks for the fix! However, I found that the sliding window in animatediff is not working with ipadapter's weight, and it seems the sub_idxs below never gets updated.

https://github.com/siliconflow/onediff/blob/main/onediff_comfy_nodes/modules/oneflow/infer_compiler_registry/register_comfy/CrossAttentionPatch.py#L108

Your environment

Ubuntu 20.04

OneDiff git commit id

2249010

OneFlow version info

libibverbs not available, ibv_fork_init skipped
path: ['/home/ubuntu/miniconda3/envs/comfy/lib/python3.10/site-packages/oneflow']
version: 0.9.1.dev20240420+cu121
git_commit: 665bcf8
cmake_build_type: Release
rdma: True
mlir: True
enterprise: False

How To Reproduce

The workflow below can reproduce the bug.
onediff_adiff_ipadapter_reproduce.json

The complete error message

No error message noted.

Additional context

Dependency version is followed this pr: #879
ComfyUI-AnimateDiff-Evolved commit id: f9e0343f4c4606ee6365a9af4a7e16118f1c45e1

The text was updated successfully, but these errors were encountered:

lijunliangTG · 2024-07-16T07:51:54Z

The bug has been successfully reproduced and the following plug-ins are required ComfyUI-VideoHelperSuite、ComfyUI-KJNodes 、ComfyUI-Custom-Scripts、ComfyUI_FizzNodes 、ComfyUI_Advanced_ControlNet The workflow is shown in the following figure：

The error message is as follows

[ERROR](GRAPH:OneflowGraph_0:OneflowGraph) building graph got error.
ERROR [2024-07-16 07:49:37] /data/home/lijunliang/project/onediff/src/onediff/infer_compiler/backends/oneflow/deployable_module.py:38 - Exception in forward: e=RuntimeError("\x1b[1m\x1b[38;2;255;000;000mError\x1b[0m: RuntimeError : This is a oneflow bug, please submit an issue at 'https://github.com/Oneflow-Inc/oneflow/issues' including the log information of the error, the minimum reproduction code, and the system information.\n")
WARNING [2024-07-16 07:49:37] /data/home/lijunliang/project/onediff/src/onediff/infer_compiler/backends/oneflow/deployable_module.py:39 - Recompile oneflow module ...
[ERROR](GRAPH:OneflowGraph_1:OneflowGraph) building graph got error.
  0%|                                                                                                     | 0/8 [00:08<?, ?it/s]
!!! Exception during processing!!! Error: RuntimeError : This is a oneflow bug, please submit an issue at 'https://github.com/Oneflow-Inc/oneflow/issues' including the log information of the error, the minimum reproduction code, and the system information.

Traceback (most recent call last):
  File "/data/home/lijunliang/project/onediff/src/onediff/infer_compiler/backends/oneflow/deployable_module.py", line 36, in wrapper
    return func(self, *args, **kwargs)
  File "/data/home/lijunliang/project/onediff/src/onediff/infer_compiler/backends/oneflow/graph_management_utils.py", line 118, in wrapper
    ret = func(self, *args, **kwargs)
  File "/data/home/lijunliang/project/onediff/src/onediff/infer_compiler/backends/oneflow/deployable_module.py", line 144, in forward
    output = dpl_graph(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/graph.py", line 295, in __call__
    self._compile(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/graph.py", line 861, in _compile
    return self._dynamic_input_graph_cache._compile(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/cache.py", line 121, in _compile
    return graph._compile(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/graph.py", line 865, in _compile
    return self._compile_new(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/graph.py", line 884, in _compile_new
    _, eager_outputs = self.build_graph(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/graph.py", line 1429, in build_graph
    outputs = self.__build_graph(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/graph.py", line 1577, in __build_graph
    outputs = self.build(*lazy_args, **lazy_kwargs)
  File "/data/home/lijunliang/project/onediff/src/onediff/infer_compiler/backends/oneflow/graph.py", line 19, in build
    return self.model(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/proxy.py", line 188, in __call__
    result = self.__block_forward(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/proxy.py", line 238, in __block_forward
    result = unbound_forward_of_module_instance(self, *args, **kwargs)
  File "/data/home/lijunliang/project/ComfyUI/custom_nodes/onediff_comfy_nodes/modules/oneflow/infer_compiler_registry/register_comfy/openaimodel.py", line 96, in forward
    h = forward_timestep_embed(
  File "/data/home/lijunliang/project/onediff/src/onediff/infer_compiler/backends/oneflow/import_tools/dyn_mock_mod.py", line 65, in proxy_func
    return sub_module()(*args, **kwargs)
  File "/data/home/lijunliang/project/ComfyUI/custom_nodes/ComfyUI_AnimateDiff_Evolved/animatediff/sampling.py", line 167, in forward_timestep_embed
    x = layer(x, context, transformer_options)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/proxy.py", line 188, in __call__
    result = self.__block_forward(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/proxy.py", line 238, in __block_forward
    result = unbound_forward_of_module_instance(self, *args, **kwargs)
  File "/data/home/lijunliang/project/ComfyUI/custom_nodes/onediff_comfy_nodes/modules/oneflow/infer_compiler_registry/register_comfy/attention.py", line 130, in forward
    x = block(x, context=context[i], transformer_options=transformer_options)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/proxy.py", line 188, in __call__
    result = self.__block_forward(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/proxy.py", line 238, in __block_forward
    result = unbound_forward_of_module_instance(self, *args, **kwargs)
  File "/data/home/lijunliang/project/ComfyUI/comfy/ldm/modules/attention.py", line 618, in forward
    n = attn2_replace_patch[block_attn2](n, context_attn2, value_attn2, extra_options)
  File "/data/home/lijunliang/project/ComfyUI/custom_nodes/onediff_comfy_nodes/modules/oneflow/infer_compiler_registry/register_comfy/CrossAttentionPatch.py", line 58, in __call__
    out = out + callback(out, q, k, v, extra_options, optimized_attention=self.optimized_attention, **self.kwargs[i], **patch_kwargs[i])
  File "/data/home/lijunliang/project/ComfyUI/custom_nodes/onediff_comfy_nodes/modules/oneflow/infer_compiler_registry/register_comfy/CrossAttentionPatch.py", line 124, in ipadapter_attention
    weight = torch.Tensor(weight[ad_params["sub_idxs"]])
RuntimeError: Error: RuntimeError : This is a oneflow bug, please submit an issue at 'https://github.com/Oneflow-Inc/oneflow/issues' including the log information of the error, the minimum reproduction code, and the system information.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/home/lijunliang/project/ComfyUI/execution.py", line 151, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
  File "/data/home/lijunliang/project/ComfyUI/execution.py", line 81, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
  File "/data/home/lijunliang/project/ComfyUI/execution.py", line 74, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
  File "/data/home/lijunliang/project/ComfyUI/nodes.py", line 1373, in sample
    return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
  File "/data/home/lijunliang/project/ComfyUI/nodes.py", line 1343, in common_ksampler
    samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
  File "/data/home/lijunliang/project/ComfyUI/custom_nodes/ComfyUI_AnimateDiff_Evolved/animatediff/sampling.py", line 520, in motion_sample
    latents = orig_comfy_sample(model, noise, *args, **kwargs)
  File "/data/home/lijunliang/project/ComfyUI/custom_nodes/ComfyUI_Advanced_ControlNet/adv_control/control_reference.py", line 47, in refcn_sample
    return orig_comfy_sample(model, *args, **kwargs)
  File "/data/home/lijunliang/project/ComfyUI/custom_nodes/ComfyUI_Advanced_ControlNet/adv_control/utils.py", line 112, in uncond_multiplier_check_cn_sample
    return orig_comfy_sample(model, *args, **kwargs)
  File "/data/home/lijunliang/project/ComfyUI/comfy/sample.py", line 43, in sample
    samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
  File "/data/home/lijunliang/project/ComfyUI/comfy/samplers.py", line 801, in sample
    return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
  File "/data/home/lijunliang/project/ComfyUI/comfy/samplers.py", line 703, in sample
    return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
  File "/data/home/lijunliang/project/ComfyUI/comfy/samplers.py", line 690, in sample
    output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
  File "/data/home/lijunliang/project/ComfyUI/comfy/samplers.py", line 669, in inner_sample
    samples = sampler.sample(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
  File "/data/home/lijunliang/project/ComfyUI/comfy/samplers.py", line 574, in sample
    samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data/home/lijunliang/project/ComfyUI/comfy/k_diffusion/sampling.py", line 160, in sample_euler_ancestral
    denoised = model(x, sigmas[i] * s_in, **extra_args)
  File "/data/home/lijunliang/project/ComfyUI/comfy/samplers.py", line 297, in __call__
    out = self.inner_model(x, sigma, model_options=model_options, seed=seed)
  File "/data/home/lijunliang/project/ComfyUI/comfy/samplers.py", line 656, in __call__
    return self.predict_noise(*args, **kwargs)
  File "/data/home/lijunliang/project/ComfyUI/comfy/samplers.py", line 659, in predict_noise
    return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed)
  File "/data/home/lijunliang/project/ComfyUI/custom_nodes/ComfyUI_AnimateDiff_Evolved/animatediff/sampling.py", line 627, in evolved_sampling_function
    cond_pred, uncond_pred = sliding_calc_conds_batch(model, [cond, uncond_], x, timestep, model_options)
  File "/data/home/lijunliang/project/ComfyUI/custom_nodes/ComfyUI_AnimateDiff_Evolved/animatediff/sampling.py", line 820, in sliding_calc_conds_batch
    sub_conds_out = calc_cond_uncond_batch_wrapper(model, sub_conds, sub_x, sub_timestep, model_options)
  File "/data/home/lijunliang/project/ComfyUI/custom_nodes/ComfyUI_AnimateDiff_Evolved/animatediff/sampling.py", line 904, in calc_cond_uncond_batch_wrapper
    return comfy.samplers.calc_cond_batch(model, conds, x_in, timestep, model_options)
  File "/data/home/lijunliang/project/ComfyUI/custom_nodes/onediff_comfy_nodes/modules/sd_hijack_utils.py", line 58, in hijacked_method
    return self(*args, **kwargs)
  File "/data/home/lijunliang/project/ComfyUI/custom_nodes/onediff_comfy_nodes/modules/sd_hijack_utils.py", line 93, in __call__
    return sub_func(self._orig_func, *args, **kwargs)
  File "/data/home/lijunliang/project/ComfyUI/custom_nodes/onediff_comfy_nodes/modules/oneflow/hijack_samplers.py", line 122, in calc_cond_batch_of
    output = model.apply_model(input_x, timestep_, **c).chunk(batch_chunks)
  File "/data/home/lijunliang/project/ComfyUI/custom_nodes/ComfyUI_Advanced_ControlNet/adv_control/utils.py", line 64, in apply_model_uncond_cleanup_wrapper
    return orig_apply_model(self, *args, **kwargs)
  File "/data/home/lijunliang/project/ComfyUI/comfy/model_base.py", line 114, in apply_model
    model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds).float()
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1582, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/data/home/lijunliang/project/onediff/src/onediff/infer_compiler/backends/oneflow/online_quantization_utils.py", line 61, in wrapper
    output = func(self, *args, **kwargs)
  File "/data/home/lijunliang/project/onediff/src/onediff/infer_compiler/backends/oneflow/args_tree_util.py", line 66, in wrapper
    output = func(self, *mapped_args, **mapped_kwargs)
  File "/data/home/lijunliang/project/onediff/src/onediff/infer_compiler/backends/oneflow/deployable_module.py", line 42, in wrapper
    return func(self, *args, **kwargs)
  File "/data/home/lijunliang/project/onediff/src/onediff/infer_compiler/backends/oneflow/graph_management_utils.py", line 121, in wrapper
    ret = func(self, *args, **kwargs)
  File "/data/home/lijunliang/project/onediff/src/onediff/infer_compiler/backends/oneflow/deployable_module.py", line 144, in forward
    output = dpl_graph(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/graph.py", line 295, in __call__
    self._compile(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/graph.py", line 861, in _compile
    return self._dynamic_input_graph_cache._compile(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/cache.py", line 121, in _compile
    return graph._compile(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/graph.py", line 865, in _compile
    return self._compile_new(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/graph.py", line 884, in _compile_new
    _, eager_outputs = self.build_graph(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/graph.py", line 1429, in build_graph
    outputs = self.__build_graph(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/graph.py", line 1577, in __build_graph
    outputs = self.build(*lazy_args, **lazy_kwargs)
  File "/data/home/lijunliang/project/onediff/src/onediff/infer_compiler/backends/oneflow/graph.py", line 19, in build
    return self.model(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/proxy.py", line 188, in __call__
    result = self.__block_forward(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/proxy.py", line 238, in __block_forward
    result = unbound_forward_of_module_instance(self, *args, **kwargs)
  File "/data/home/lijunliang/project/ComfyUI/custom_nodes/onediff_comfy_nodes/modules/oneflow/infer_compiler_registry/register_comfy/openaimodel.py", line 96, in forward
    h = forward_timestep_embed(
  File "/data/home/lijunliang/project/onediff/src/onediff/infer_compiler/backends/oneflow/import_tools/dyn_mock_mod.py", line 65, in proxy_func
    return sub_module()(*args, **kwargs)
  File "/data/home/lijunliang/project/ComfyUI/custom_nodes/ComfyUI_AnimateDiff_Evolved/animatediff/sampling.py", line 167, in forward_timestep_embed
    x = layer(x, context, transformer_options)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/proxy.py", line 188, in __call__
    result = self.__block_forward(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/proxy.py", line 238, in __block_forward
    result = unbound_forward_of_module_instance(self, *args, **kwargs)
  File "/data/home/lijunliang/project/ComfyUI/custom_nodes/onediff_comfy_nodes/modules/oneflow/infer_compiler_registry/register_comfy/attention.py", line 130, in forward
    x = block(x, context=context[i], transformer_options=transformer_options)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/proxy.py", line 188, in __call__
    result = self.__block_forward(*args, **kwargs)
  File "/data/home/lijunliang/anaconda3/envs/sd2/lib/python3.10/site-packages/oneflow/nn/graph/proxy.py", line 238, in __block_forward
    result = unbound_forward_of_module_instance(self, *args, **kwargs)
  File "/data/home/lijunliang/project/ComfyUI/comfy/ldm/modules/attention.py", line 618, in forward
    n = attn2_replace_patch[block_attn2](n, context_attn2, value_attn2, extra_options)
  File "/data/home/lijunliang/project/ComfyUI/custom_nodes/onediff_comfy_nodes/modules/oneflow/infer_compiler_registry/register_comfy/CrossAttentionPatch.py", line 58, in __call__
    out = out + callback(out, q, k, v, extra_options, optimized_attention=self.optimized_attention, **self.kwargs[i], **patch_kwargs[i])
  File "/data/home/lijunliang/project/ComfyUI/custom_nodes/onediff_comfy_nodes/modules/oneflow/infer_compiler_registry/register_comfy/CrossAttentionPatch.py", line 124, in ipadapter_attention
    weight = torch.Tensor(weight[ad_params["sub_idxs"]])
RuntimeError: Error: RuntimeError : This is a oneflow bug, please submit an issue at 'https://github.com/Oneflow-Inc/oneflow/issues' including the log information of the error, the minimum reproduction code, and the system information.

Currently trying to find the minimum recurrence code

lijunliangTG · 2024-07-22T02:07:42Z

Now you can switch to the fix_comfy_get_weight_subidxs branch, which will be merged into the main branch

@ccssu

…ode (#1020) Issue 904 does not execute properly in graph mode，```weight = torch.Tensor(weight[ad_params["sub_idxs"]])```Because the above code only indexes and reassigns values at run time, this dynamic operation can result in static graph build failures or run-time errors. fixed #904 感谢 @ccssu 对本次PR的帮助。

Gutianpei assigned strint May 23, 2024

strint assigned ccssu Jul 5, 2024

strint added the sig-comfyui label Jul 5, 2024

lijunliangTG mentioned this issue Jul 18, 2024

fix Dynamic Indexing and Assignment Operations Fail in Static Graph Mode #1020

Merged

lijunliangTG closed this as completed in #1020 Jul 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IPAdapter + Sliding window in animatediff is not working #904

IPAdapter + Sliding window in animatediff is not working #904

Gutianpei commented May 23, 2024

lijunliangTG commented Jul 16, 2024 •

edited

Loading

lijunliangTG commented Jul 22, 2024

IPAdapter + Sliding window in animatediff is not working #904

IPAdapter + Sliding window in animatediff is not working #904

Comments

Gutianpei commented May 23, 2024

Describe the bug

Your environment

OneDiff git commit id

OneFlow version info

How To Reproduce

The complete error message

Additional context

lijunliangTG commented Jul 16, 2024 • edited Loading

lijunliangTG commented Jul 22, 2024

lijunliangTG commented Jul 16, 2024 •

edited

Loading