StableDiffusionControlNetInpaintPipeline bug in nexfort backend

[zhangvia]

Describe the bug

A clear and concise description of what the bug is.

使用nexfort后端编译时出现如下报错：

/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:06<00:00,  1.21s/it]
You have disabled the safety checker for <class 'diffusers.pipelines.controlnet.pipeline_controlnet_inpaint.StableDiffusionControlNetInpaintPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances, disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
  0%|                                                                                                                                                                  | 0/50 [00:08<?, ?it/s]
Traceback (most recent call last):
  File "/media/74nvme/research/arcsoft_model_fantasy/test.py", line 172, in <module>
    pipe(
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint.py", line 1442, in __call__
    down_block_res_samples, mid_block_res_sample = self.controlnet(
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/onediff/infer_compiler/backends/nexfort/deployable_module.py", line 22, in forward
    return self._deployable_module_model(*args, **kwargs)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 451, in _fn
    return fn(*args, **kwargs)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 921, in catch_errors
    return callback(frame, cache_entry, hooks, frame_state, skip=1)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 786, in _convert_frame
    result = inner_convert(
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 400, in _convert_frame_assert
    return _compile(
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 676, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 262, in time_wrapper
    r = func(*args, **kwargs)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 535, in compile_inner
    out_code = transform_code_object(code, transform)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1036, in transform_code_object
    transformations(instructions, code_options)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 165, in _fn
    return fn(*args, **kwargs)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 500, in transform
    tracer.run()
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2149, in run
    super().run()
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 810, in run
    and self.step()
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 773, in step
    getattr(self, inst.opname)(inst)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2268, in RETURN_VALUE
    self.output.compile_subgraph(
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 991, in compile_subgraph
    self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1168, in compile_and_call_fx_graph
    compiled_fn = self.call_user_compiler(gm)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 262, in time_wrapper
    r = func(*args, **kwargs)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1241, in call_user_compiler
    raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1222, in call_user_compiler
    compiled_fn = compiler_fn(gm, self.example_inputs())
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
    compiled_gm = compiler_fn(gm, example_inputs)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/__init__.py", line 1768, in __call__
    return self.compiler_fn(model_, inputs_, **self.kwargs)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/nexfort/dynamo/backends/nexfort.py", line 16, in nexfort
    return compile_fx(gm, example_inputs, mode=mode, options=options)
  File "src/nexfort/fx_compiler/fx_compiler.py", line 49, in nexfort.fx_compiler.fx_compiler.compile_fx
  File "src/nexfort/fx_compiler/fx_compiler.py", line 91, in nexfort.fx_compiler.fx_compiler.compile_fx_inner
  File "src/nexfort/fx_compiler/fx_compiler.py", line 91, in nexfort.fx_compiler.fx_compiler.compile_fx_inner
  File "src/nexfort/fx_compiler/fx_compiler.py", line 114, in nexfort.fx_compiler.fx_compiler.compile_fx_inner
  File "src/nexfort/fx_compiler/overrides.py", line 72, in nexfort.fx_compiler.overrides.with_override_torch_env.decorator.wrapper
  File "src/nexfort/fx_compiler/overrides.py", line 73, in nexfort.fx_compiler.overrides.with_override_torch_env.decorator.wrapper
  File "src/nexfort/fx_compiler/fx_compiler.py", line 319, in nexfort.fx_compiler.fx_compiler.fx_compile
  File "src/nexfort/fx_compiler/fx_compiler.py", line 327, in nexfort.fx_compiler.fx_compiler.fx_compile
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/__init__.py", line 1729, in __call__
    return compile_fx(model_, inputs_, config_patches=self.config)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1330, in compile_fx
    return aot_autograd(
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 58, in compiler_fn
    cg = aot_module_simplified(gm, example_inputs, **kwargs)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 903, in aot_module_simplified
    compiled_fn = create_aot_dispatcher_function(
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 262, in time_wrapper
    r = func(*args, **kwargs)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 533, in create_aot_dispatcher_function
    fw_metadata = run_functionalized_fw_and_collect_metadata(
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/collect_metadata_analysis.py", line 112, in inner
    flat_f_outs = f(*flat_f_args)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/traced_function_transforms.py", line 676, in functional_call
    out = PropagateUnbackedSymInts(mod).run(
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/fx/interpreter.py", line 145, in run
    self.env[node] = self.run_node(node)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/traced_function_transforms.py", line 651, in run_node
    result = super().run_node(n)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/fx/interpreter.py", line 202, in run_node
    return getattr(self, n.op)(n.target, args, kwargs)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/fx/interpreter.py", line 274, in call_function
    return target(*args, **kwargs)
  File "/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/torch/_ops.py", line 594, in __call__
    return self_._op(*args, **kwargs)
torch._dynamo.exc.BackendCompilerFailed: backend='nexfort' raised:
RuntimeError: aten::cat() Expected a value of type 'List[Tensor]' for argument 'tensors' but instead found type 'immutable_list'.
Position: 0
Value: [None, None, None]
Declaration: aten::cat(Tensor[] tensors, int dim=0) -> Tensor
Cast error details: Unable to cast Python instance of type <class 'torch.fx.immutable_collections.immutable_list'> to C++ type '?' (#define PYBIND11_DETAILED_ERROR_MESSAGES or compile in debug mode for details)

While executing %cat_default_1 : [num_users=1] = call_function[target=torch.ops.aten.cat.default](args = ([None, None, None], 0), kwargs = {})
Original traceback:
None

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information


You can suppress this exception and fall back to eager by setting:
    import torch._dynamo
    torch._dynamo.config.suppress_errors = True

Your environment

OS

OneDiff git commit id

Name: onediff
Version: 1.2.0.dev202406240130
Summary: an out-of-the-box acceleration library for diffusion models
Home-page: https://github.com/siliconflow/onediff
Author: OneDiff contributors
Author-email: contact@siliconflow.com
License: Apache-2.0
Location: /media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages
Requires: accelerate, diffusers, torch, transformers
Required-by:

OneFlow version info if you have installed oneflow

Run python -m oneflow --doctor and paste it here.

libibverbs not available, ibv_fork_init skipped
W20240708 05:56:45.974316 22080 cuda_device_descriptor.cpp:91] The CUDA device 'NVIDIA TITAN Xp' with capability 61 is not compatible with the current OneFlow installation. The current program may throw a 'no kernel image is available for execution on the device' error or hang for a long time. Please reinstall OneFlow compiled with a newer version of CUDA.
path: ['/media/74nvme/software/miniconda3/envs/stable-fast/lib/python3.10/site-packages/oneflow']
version: 0.9.1.dev20240413+cu118
git_commit: 67341c9
cmake_build_type: Release
rdma: True
mlir: True
enterprise: False

How To Reproduce

Steps to reproduce the behavior(code or script):
代码如下：

import torch
from diffusers import StableDiffusionControlNetInpaintPipeline,ControlNetModel
from onediff.infer_compiler import oneflow_compile,compile
from diffusers.utils import load_image
from os.path import join as opj
import cv2
import numpy as np
from PIL import Image

model_path = "/media/74nvme/research/models/"
model_path_base = "/media/74nvme/research/models"
canny_controlnet_path = opj(model_path_base,"controlnet/control_v11p_sd15_canny")

# cn_path = [canny_controlnet_path,depth_controlnet_path,hand_controlnet_path,tile_controlnet_path]
cn_path = [canny_controlnet_path]
controlnets = []


for i in cn_path:
  controlnets.append(ControlNetModel.from_pretrained(i,torch_dtype=torch.float16))

image = Image.open("/media/74nvme/research/test_img/805565039-0.png").resize((512,768))
sd_mask_image = Image.open("/media/74nvme/research/test_img/805565039-0-mask.jpg").resize((512,768))
controlnet_conditioning_image = [Image.fromarray(cv2.Canny(np.array(image),100,200))]
ip_adapter_image = None


pipe = StableDiffusionControlNetInpaintPipeline.from_pretrained(model_path, controlnet=controlnets, torch_dtype=torch.float16).to('cuda')
# pipe.load_ip_adapter(opj(model_path_base,"controlnet/ip_adapter/"), subfolder="./", weight_name="ip-adapter_sd15.bin")
# pipe.set_ip_adapter_scale(0.9)
pipe.unet = compile(pipe.unet,backend='nexfort')
# pipe.text_encoder = oneflow_compile(pipe.text_encoder)
# pipe.controlnet = oneflow_compile(pipe.controlnet)
pipe.controlnet = compile(pipe.controlnet,backend='nexfort')

# pipe.set_ip_adapter_scale(0)
pipe(
  prompt="a girl",
  image=image.resize((1024,744)),
  mask_image=sd_mask_image.resize((1024,744)),
  control_image=[image.resize((1024,744)) for image in controlnet_conditioning_image],
  ip_adapter_image=ip_adapter_image.resize((1024,744)) if ip_adapter_image is not None else None,
  output_type="pil",
  controlnet_conditioning_scale=[0.5]
)
# image.save("./res2_nocompile.jpg")

The complete error message

Additional context

Add any other context about the problem here.

[strint]

Thanks, Got it. @zhangvia

nexfort backend is beta release and only tested under sd3 and pix art.

After we finish SD series model(#998) support we will get back to this. @marigoold

[marigoold]

Hi @zhangvia
This bug has been fixed in the latest Nexfort version. Please wait for the upcoming nightly release and then update Nexfort.

[zhangvia]

confirmed，the bug is fixed in version nexfort 0.1.dev254. but nexfort doesn't support ipadapter

[strint]

confirmed，the bug is fixed in version nexfort 0.1.dev254.

cool

but nexfort doesn't support ipadapter

nexfort ipadapter is on the roadmap