add xpu device check before moving tensor directly to xpu device

[faaany]

What does this PR do?

The following code snippet will throw an error on XPU

import transformers
import torch

model_id = "meta-llama/Llama-2-7b-chat-hf"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="xpu"
)

Traceback (most recent call last):
  File "/workspace1/fanli/workspace/test.py", line 6, in <module>
    pipeline = transformers.pipeline(
  File "/home/fanli/miniforge3/envs/ds-ut-ww26/lib/python3.10/site-packages/transformers/pipelines/__init__.py", line 895, in pipeline
    framework, model = infer_framework_load_model(
  File "/home/fanli/miniforge3/envs/ds-ut-ww26/lib/python3.10/site-packages/transformers/pipelines/base.py", line 283, in infer_framework_load_model
    model = model_class.from_pretrained(model, **kwargs)
  File "/home/fanli/miniforge3/envs/ds-ut-ww26/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
  File "/home/fanli/miniforge3/envs/ds-ut-ww26/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3838, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/home/fanli/miniforge3/envs/ds-ut-ww26/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4298, in _load_pretrained_model
    new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
  File "/home/fanli/miniforge3/envs/ds-ut-ww26/lib/python3.10/site-packages/transformers/modeling_utils.py", line 895, in _load_state_dict_into_meta_model
    set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs)
  File "/workspace1/fanli/workspace/accelerate/src/accelerate/utils/modeling.py", line 410, in set_module_tensor_to_device
    new_value = value.to(device)
NotImplementedError: Could not run 'aten::empty_strided' with arguments from the 'XPU' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::empty_strided' is only available for these backends: [CPU, Meta, QuantizedCPU, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradHIP, AutogradXLA, AutogradMPS, AutogradIPU, AutogradXPU, AutogradHPU, AutogradVE, AutogradLazy, AutogradMTIA, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, AutogradMeta, AutogradNestedTensor, Tracer, AutocastCPU, AutocastCUDA, FuncTorchBatched, BatchedNestedTensor, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PreDispatch, PythonDispatcher].

similar to the check here in transformers.pipeline.

[faaany]

@yao-matrix

[yao-matrix]

OK for me.

[faaany]

@muellerzr and @SunMarc pls have a review, thx a lot!

[SunMarc]

Thanks for fixing @faaany ! Left a comment !

[muellerzr]

Okay to me as well!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[SunMarc]

Nice !