Segmentation fault with inductor for huggingface/diffusers

[mreso]

Hi,

running the stable_diffusion example from huggingface/diffusers results in segmentation fault.

Repro:
pip install --upgrade diffusers transformers

Then run:

import torchdynamo

from diffusers import StableDiffusionPipeline

pipeline = DiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4") #ACCESS TOKEN REQUIRED
pipe = pipe.to("cuda")

@torchdynamo.optimize("inductor")
def apply(x):
    return pipe(x)

prompt = "a photo of an astronaut riding a horse on mars"
image = apply(prompt).images[0]  

Error message:

ftfy or spacy is not installed using BERT BasicTokenizer instead of ftfy.
WARNING:torchinductor.lowering:make_fallback(aten.unfold): a decomposition exists, we should switch to it
WARNING:torchinductor.lowering:make_fallback(aten.unfold_backward): a decomposition exists, we should switch to it
[2022-10-15 01:43:28,669] torchdynamo.symbolic_convert: [WARNING] Graph break: call_function UserDefinedObjectVariable(StableDiffusionPipeline) [ConstantVariable(str)] {} from user code at   File "/home/ubuntu/torchdynamo/test.py", line 14, in apply
    return pipe(x)

[2022-10-15 01:43:28,770] torchdynamo.symbolic_convert: [WARNING] Graph break: call_function UserDefinedObjectVariable(CLIPTokenizer) [ConstantVariable(str)] {'padding': ConstantVariable(str), 'max_length': ConstantVariable(int), 'return_tensors': Cons
tantVariable(str)} from user code at   File "/home/ubuntu/miniconda3/envs/torchdynamo/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 214, in __call__
    text_inputs = self.tokenizer(

[2022-10-15 01:43:34,247] torchdynamo.symbolic_convert: [WARNING] Graph break: call_function BuiltinVariable(dict) [UserDefinedObjectVariable(FrozenDict)] {} from user code at   File "/home/ubuntu/miniconda3/envs/torchdynamo/lib/python3.10/site-package
s/diffusers/pipeline_utils.py", line 205, in device
    module_names, _ = self.extract_init_dict(dict(self.config))

[2022-10-15 01:43:37,990] torchinductor.graph: [WARNING] Creating implicit fallback for:
  target: aten.triu_.default
  args[0]: TensorBox(StorageBox(
    ComputedBuffer(name='buf0', layout=FlexibleLayout('cpu', torch.float32, size=[1, 77, 77], stride=[5929, 77, 1]), data=Pointwise(
      'cpu',
      torch.float32,
      constant(-3.4028234663852886e+38, torch.float32),
      ranges=[1, 77, 77],
      origins={fill_}
    ))
  ))
  args[1]: 1
[2022-10-15 01:43:37,992] torchinductor.ir: [WARNING] Using FallbackKernel: torch.ops.aten.triu_.default
[2022-10-15 01:43:37,992] torchinductor.ir: [WARNING] DeviceCopy
[2022-10-15 01:43:43,374] torchinductor.compile_fx: [WARNING] skipping cudagraphs due to multiple devices
[2022-10-15 01:43:55,938] torchdynamo.symbolic_convert: [WARNING] Graph break: call_method GetAttrVariable(UserDefinedObjectVariable(Version), _regex) search [ConstantVariable(str)] {} from user code at   File "/home/ubuntu/miniconda3/envs/torchdynamo/
lib/python3.10/site-packages/diffusers/schedulers/scheduling_pndm.py", line 161, in set_timesteps
    deprecated_offset = deprecate(
  File "/home/ubuntu/miniconda3/envs/torchdynamo/lib/python3.10/site-packages/diffusers/utils/deprecation_utils.py", line 17, in deprecate
    if version.parse(version.parse(__version__).base_version) >= version.parse(version_name):
  File "/home/ubuntu/miniconda3/envs/torchdynamo/lib/python3.10/site-packages/packaging/version.py", line 49, in parse
    return Version(version)
  File "/home/ubuntu/miniconda3/envs/torchdynamo/lib/python3.10/site-packages/packaging/version.py", line 264, in __init__
    match = self._regex.search(version)

[2022-10-15 01:43:55,947] torchdynamo.symbolic_convert: [WARNING] Graph break: numpy from user code at   File "/home/ubuntu/miniconda3/envs/torchdynamo/lib/python3.10/site-packages/diffusers/schedulers/scheduling_pndm.py", line 170, in <graph break in
set_timesteps>
    self._timesteps = (np.arange(0, num_inference_steps) * step_ratio).round()

[2022-10-15 01:43:56,134] torchdynamo.symbolic_convert: [WARNING] Graph break: call_function in skip_files /home/ubuntu/miniconda3/envs/torchdynamo/lib/python3.10/site-packages/tqdm/asyncio.py from user code at   File "/home/ubuntu/miniconda3/envs/torc
hdynamo/lib/python3.10/site-packages/diffusers/pipeline_utils.py", line 559, in progress_bar
    return tqdm(iterable, **self._progress_bar_config)

  0%|
                                                                                       | 0/51 [00:00<?, ?it/s][2022-10-15 01:43:59,438] torchdynamo.symbolic_convert: [WARNING] Graph break: call_function UserDefinedClassVariable() [] {'sample': TensorVa
riable()} from user code at   File "/home/ubuntu/miniconda3/envs/torchdynamo/lib/python3.10/site-packages/diffusers/models/unet_2d_condition.py", line 341, in forward
    return UNet2DConditionOutput(sample=sample)

[2022-10-15 01:47:47,542] torchdynamo.symbolic_convert: [WARNING] Graph break: inline __setitem__ from user code at   File "<string>", line 4, in <graph break in __init__>
  File "/home/ubuntu/miniconda3/envs/torchdynamo/lib/python3.10/site-packages/diffusers/utils/outputs.py", line 72, in __post_init__
    self[field.name] = v

[2022-10-15 01:47:47,550] torchdynamo.symbolic_convert: [WARNING] Graph break: non-function or method super: <slot wrapper '__setitem__' of 'collections.OrderedDict' objects> from user code at   File "/home/ubuntu/miniconda3/envs/torchdynamo/lib/python
3.10/site-packages/diffusers/utils/outputs.py", line 104, in __setitem__
    super().__setitem__(key, value)

[2022-10-15 01:47:47,553] torchdynamo.symbolic_convert: [WARNING] Graph break: non-function or method super: <slot wrapper '__setattr__' of 'object' objects> from user code at   File "/home/ubuntu/miniconda3/envs/torchdynamo/lib/python3.10/site-package
s/diffusers/utils/outputs.py", line 106, in <graph break in __setitem__>
    super().__setattr__(key, value)

[2022-10-15 01:47:47,565] torchdynamo.symbolic_convert: [WARNING] Graph break: data dependent operator: aten._local_scalar_dense.default from user code at   File "/home/ubuntu/miniconda3/envs/torchdynamo/lib/python3.10/site-packages/diffusers/scheduler
s/scheduling_pndm.py", line 225, in step
    return self.step_plms(model_output=model_output, timestep=timestep, sample=sample, return_dict=return_dict)
  File "/home/ubuntu/miniconda3/envs/torchdynamo/lib/python3.10/site-packages/diffusers/schedulers/scheduling_pndm.py", line 340, in step_plms
    prev_sample = self._get_prev_sample(sample, timestep, prev_timestep, model_output)
  File "/home/ubuntu/miniconda3/envs/torchdynamo/lib/python3.10/site-packages/diffusers/schedulers/scheduling_pndm.py", line 374, in _get_prev_sample
    alpha_prod_t = self.alphas_cumprod[timestep]

[2022-10-15 01:47:48,531] torchdynamo.symbolic_convert: [WARNING] Graph break: data dependent operator: aten._local_scalar_dense.default from user code at   File "/home/ubuntu/miniconda3/envs/torchdynamo/lib/python3.10/site-packages/diffusers/scheduler
s/scheduling_pndm.py", line 375, in <graph break in _get_prev_sample>
    alpha_prod_t_prev = self.alphas_cumprod[prev_timestep] if prev_timestep >= 0 else self.final_alpha_cumprod

[2022-10-15 01:47:49,100] torchinductor.compile_fx: [WARNING] skipping cudagraphs due to multiple devices
Segmentation fault (core dumped)

Expected result:
Example works with eager + aot_eager and successfully generates an image. Crashed with inductor.

Env info:

Collecting environment information...
PyTorch version: 1.14.0.dev20221014+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.24.1
Libc version: glibc-2.31

Python version: 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:36:39) [GCC 10.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-1020-aws-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A10G
Nvidia driver version: 510.73.08
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.4
[pip3] torch==1.14.0.dev20221014+cu116
[pip3] torchdynamo==1.14.0.dev0
[conda] numpy                     1.23.4                   pypi_0    pypi
[conda] torch                     1.14.0.dev20221014+cu116          pypi_0    pypi
[conda] torchdynamo               1.14.0.dev0              pypi_0    pypi

[voznesenskym]

Can you run the minifier on this please?

[anijain2305]

@mreso These are my steps to make progress

• I ran it with eager, aot_eager backend. Both were passing. inductor failed with segfault.
• I ran with TORCHDYNAMO_REPRO_AFTER="aot" but nothing happened. And this was my mistake, I forgot to tell that for segfaults, we need to add one more flag. TORCHDYNAMO_REPRO_AFTER="aot" TORCHDYNAMO_REPRO_LEVEL=3
• With this, I got a small minifier_launcher.py. This minifier launcher got hung ... sigh. I will make changes so that either it does not hang or gracefully exits.
• I looked at minifier_launcher.py and manually added code to run inductor. And was able to get a segfault.

cc @mlazos for the debugging playbook edits.

This is the repro

import torch
from torch import tensor, device
import torch.fx as fx
from torch._dynamo.testing import rand_strided
from math import inf
from torch.fx.experimental.proxy_tensor import make_fx

# torch version: 1.14.0a0+git8a8cd09
# torch cuda version: 11.6
# torch git version: 8a8cd092c8537b226f5c38ed88bc07e181b0946c


# CUDA Info:
# nvcc: NVIDIA (R) Cuda compiler driver
# Copyright (c) 2005-2022 NVIDIA Corporation
# Built on Thu_Feb_10_18:23:41_PST_2022
# Cuda compilation tools, release 11.6, V11.6.112
# Build cuda_11.6.r11.6/compiler.30978841_0

# GPU Hardware Info:
# NVIDIA A100-SXM4-40GB : 2


from torch.nn import *
from torch._inductor.compile_fx import compile_fx_inner

class Repro(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.register_buffer('_tensor_constant0', torch.randn([], dtype=torch.float32))
        self.register_buffer('_tensor_constant1', torch.randn([], dtype=torch.float32))



    def forward(self, arg0_1, arg1_1, arg2_1, arg3_1):
        _tensor_constant0 = self._tensor_constant0
        lift_fresh_copy = torch.ops.aten.lift_fresh_copy.default(_tensor_constant0);  _tensor_constant0 = None
        sub = torch.ops.aten.sub.Tensor(lift_fresh_copy, arg3_1);  lift_fresh_copy = None
        _tensor_constant1 = self._tensor_constant1
        lift_fresh_copy_1 = torch.ops.aten.lift_fresh_copy.default(_tensor_constant1);  _tensor_constant1 = None
        sub_1 = torch.ops.aten.sub.Tensor(lift_fresh_copy_1, arg0_1);  lift_fresh_copy_1 = None
        div = torch.ops.aten.div.Tensor(arg0_1, arg3_1)
        pow_1 = torch.ops.aten.pow.Tensor_Scalar(div, 0.5);  div = None
        pow_2 = torch.ops.aten.pow.Tensor_Scalar(sub_1, 0.5);  sub_1 = None
        mul = torch.ops.aten.mul.Tensor(arg3_1, pow_2);  pow_2 = None
        mul_1 = torch.ops.aten.mul.Tensor(arg3_1, sub);  sub = None
        mul_2 = torch.ops.aten.mul.Tensor(mul_1, arg0_1);  mul_1 = None
        pow_3 = torch.ops.aten.pow.Tensor_Scalar(mul_2, 0.5);  mul_2 = None
        add = torch.ops.aten.add.Tensor(mul, pow_3);  mul = pow_3 = None
        mul_3 = torch.ops.aten.mul.Tensor(pow_1, arg1_1);  pow_1 = arg1_1 = None
        sub_2 = torch.ops.aten.sub.Tensor(arg0_1, arg3_1);  arg0_1 = arg3_1 = None
        mul_4 = torch.ops.aten.mul.Tensor(sub_2, arg2_1);  sub_2 = arg2_1 = None
        div_1 = torch.ops.aten.div.Tensor(mul_4, add);  mul_4 = add = None
        sub_3 = torch.ops.aten.sub.Tensor(mul_3, div_1);  mul_3 = div_1 = None
        return (sub_3,)

args = [((), (), torch.float32, 'cpu'), ((1, 4, 64, 64), (16384, 4096, 64, 1), torch.float32, 'cuda'), ((1, 4, 64, 64), (16384, 4096, 64, 1), torch.float32, 'cuda'), ((), (), torch.float32, 'cuda')]
args = [rand_strided(sh, st, dt, dev) for (sh, st, dt, dev) in args]
mod = make_fx(Repro().to(device="cuda"))(*args)

opt_mod = compile_fx_inner(mod, args)
opt_mod(args)

[anijain2305]

cc @ngimel @Chillee for the actual segfault with inductor.

[ngimel]

Codegen generates buf0 on cpu and passes it to triton kernel:

buf0 = empty_strided((1, s0, s1, s1), (s0*(s1*s1), s1*s1, s1, 1), device='cpu', dtype=torch.float32)

Also, kernel args are sus, never seen c_void_p in the generated code

kernel0(c_void_p(arg0_1.data_ptr()), c_void_p(arg3_1.data_ptr()), c_void_p(arg1_1.data_ptr()), c_void_p(arg2_1.data_ptr()), c_void_p(buf0.data_ptr()), c_long(s0), c_long(s1))

[ngimel]

@yanboliang we were supposed to copy all 0d cpu args to gpu? arg0_1 is on cpu

[ngimel]

0d scalar handling is dead wrong (why didn't OpInfo catch this? Probably we are not running all samples yet).

@torchdynamo.optimize("inductor")
def div(x, y):
    return torch.ops.aten.div(x, y)

x = torch.tensor(3., dtype=torch.float, device="cpu")
y = torch.randn(4, dtype=torch.float, device="cuda")
print(div(x,y))

[penguinwu]

"I ran with TORCHDYNAMO_REPRO_AFTER="aot" but nothing happened. And this was my mistake, I forgot to tell that for segfaults, we need to add one more flag. TORCHDYNAMO_REPRO_AFTER="aot" TORCHDYNAMO_REPRO_LEVEL=4"

@mlazos Could the rageshaker tool be smart enough to know to enable TORCHDYNAMO_REPRO_LEVEL=4 based on error types?

[yanboliang]

@yanboliang we were supposed to copy all 0d cpu args to gpu? arg0_1 is on cpu

Yes, that's what we did now. We can't tell the if the 0d tensor is converted from unspecialized variable or user's explicit input at inductor level.

[mlazos]

"I ran with TORCHDYNAMO_REPRO_AFTER="aot" but nothing happened. And this was my mistake, I forgot to tell that for segfaults, we need to add one more flag. TORCHDYNAMO_REPRO_AFTER="aot" TORCHDYNAMO_REPRO_LEVEL=4"

@mlazos Could the rageshaker tool be smart enough to know to enable TORCHDYNAMO_REPRO_LEVEL=4 based on error types?

I'm not sure how segfaults are detected, if the whole program (with dynamo) crashes, I'm not sure how the minifier can be triggered after this event. I do think we can automatically disambiguate with the other error levels though.

[anijain2305]

Another segfault related to 0d tensors. This one affects longformer models

import torch
from torch import tensor, device
import torch.fx as fx
from torch._dynamo.testing import rand_strided
from math import inf
from torch.fx.experimental.proxy_tensor import make_fx

# torch version: 1.14.0a0+git8a8cd09
# torch cuda version: 11.6
# torch git version: 8a8cd092c8537b226f5c38ed88bc07e181b0946c


# CUDA Info:
# nvcc: NVIDIA (R) Cuda compiler driver
# Copyright (c) 2005-2022 NVIDIA Corporation
# Built on Thu_Feb_10_18:23:41_PST_2022
# Cuda compilation tools, release 11.6, V11.6.112
# Build cuda_11.6.r11.6/compiler.30978841_0

# GPU Hardware Info:
# NVIDIA A100-SXM4-40GB : 2


from torch.nn import *
class Repro(torch.nn.Module):
    def __init__(self):
        super().__init__()



    def forward(self, primals_1, primals_2):
        unsqueeze = torch.ops.aten.unsqueeze.default(primals_1, 4);  primals_1 = None
        permute = torch.ops.aten.permute.default(unsqueeze, [0, 1, 2, 4, 3])
        permute_1 = torch.ops.aten.permute.default(unsqueeze, [0, 1, 4, 2, 3]);  unsqueeze = None
        permute_2 = torch.ops.aten.permute.default(permute, [0, 1, 2, 4, 3]);  permute = None
        clone = torch.ops.aten.clone.default(permute_2, memory_format = torch.contiguous_format);  permute_2 = None
        _unsafe_view = torch.ops.aten._unsafe_view.default(clone, [36, 512, 64]);  clone = None
        permute_3 = torch.ops.aten.permute.default(permute_1, [0, 1, 4, 3, 2]);  permute_1 = None
        clone_1 = torch.ops.aten.clone.default(permute_3, memory_format = torch.contiguous_format);  permute_3 = None
        _unsafe_view_1 = torch.ops.aten._unsafe_view.default(clone_1, [36, 64, 512]);  clone_1 = None
        bmm = torch.ops.aten.bmm.default(_unsafe_view, _unsafe_view_1)
        view = torch.ops.aten.view.default(bmm, [12, 3, 512, 1, 512]);  bmm = None
        permute_4 = torch.ops.aten.permute.default(view, [0, 1, 2, 4, 3]);  view = None
        sum_1 = torch.ops.aten.sum.dim_IntList(permute_4, [4]);  permute_4 = None
        constant_pad_nd = torch.ops.aten.constant_pad_nd.default(sum_1, [0, 0, 0, 1], 0.0);  sum_1 = None
        view_1 = torch.ops.aten.view.default(constant_pad_nd, [12, 3, 512, 513]);  constant_pad_nd = None
        add = torch.ops.aten.add.Tensor(primals_2, 1);  primals_2 = None
        permute_6 = torch.ops.aten.permute.default(_unsafe_view, [0, 2, 1]);  _unsafe_view = None
        permute_7 = torch.ops.aten.permute.default(_unsafe_view_1, [0, 2, 1]);  _unsafe_view_1 = None
        return [view_1, add, permute_6, permute_7]

args = [((12, 3, 512, 64), (64, 196608, 768, 1), torch.float32, 'cuda'), ((), (), torch.int64, 'cpu')]
args = [rand_strided(sh, st, dt, dev) for (sh, st, dt, dev) in args]
mod = make_fx(Repro().to(device="cuda"))(*args)

from torch._inductor.compile_fx import compile_fx_inner
opt_mod = compile_fx_inner(mod, args)
opt_mod(args)