[pipelines] add better checking when a wrong model is passed when initializing a pipeline

[sayakpaul]

With this code:

from diffusers import DiffusionPipeline 
from transformers import T5EncoderModel 
import torch 

repo_id = "black-forest-labs/FLUX.1-dev"
text_encoder_2 = T5EncoderModel.from_pretrained(repo_id, subfolder="text_encoder_2")
pipe = DiffusionPipeline.from_pretrained(
    repo_id, text_encoder=text_encoder_2, torch_dtype=torch.bfloat16
).to("cuda")

image = pipe("A dog").images[0]

where we're doing text_encoder=text_encoder_2 (should have been text_encoder_2=text_encoder_2), we get an extremely uninformative error:

Error

...
... [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [95,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
  File "/home/sayak/diffusers/check_incompatible_flux.py", line 11, in <module>
    image = pipe("A dog").images[0]
  File "/home/sayak/.pyenv/versions/3.10.12/envs/diffusers/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/home/sayak/diffusers/src/diffusers/pipelines/flux/pipeline_flux.py", line 677, in __call__
    ) = self.encode_prompt(
  File "/home/sayak/diffusers/src/diffusers/pipelines/flux/pipeline_flux.py", line 353, in encode_prompt
    pooled_prompt_embeds = self._get_clip_prompt_embeds(
  File "/home/sayak/diffusers/src/diffusers/pipelines/flux/pipeline_flux.py", line 289, in _get_clip_prompt_embeds
    prompt_embeds = self.text_encoder(text_input_ids.to(device), output_hidden_states=False)
  File "/home/sayak/.pyenv/versions/3.10.12/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/sayak/.pyenv/versions/3.10.12/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/sayak/.pyenv/versions/3.10.12/envs/diffusers/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1972, in forward
    encoder_outputs = self.encoder(
  File "/home/sayak/.pyenv/versions/3.10.12/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/sayak/.pyenv/versions/3.10.12/envs/diffusers/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/sayak/.pyenv/versions/3.10.12/envs/diffusers/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1029, in forward
    attention_mask = torch.ones(batch_size, mask_seq_length, device=inputs_embeds.device)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

I know it's on the users to ensure that they're initializing a pipeline properly with the right components. However, the thrown error makes it worse to do any further debugging on the user side. IMO.

It would be greatly helpful from the perspective of a user if we could throw a better warning/error instead. As soon as we detect wrong types (I think model_index.json info can be used in this case), we throw a warning/error. Is this something doable?

Cc: @DN6 @yiyixuxu
Ccing @hlky as well in case you're interested.

[kameshkanna]

@sayakpaul

1. adding a validate_pipeline() method in pipeline loading class if not validated throw simple yet understandable errors like these not sure how good these are:

`ValueError: Mismatched model type for component 'text_encoder_2'. 
Expected types: ['T5EncoderModel'], but got: 'T5Model' 

Ensure the component matches the expected type as specified in the `model_index.json``

2. While using model_index.json() If a mismatch is detected, throw a clear and actionable error, if strict validation isn’t ideal, show warnings instead of hard errors.