Summary pipeline error when gpu enabled on mps device

[Vincent-liuwingsang]

On Apple silicon mps device, default setting for Summary errors with the following
AssertionError: Torch not compiled with CUDA enabled

from txtai.pipeline import Summary
summary = Summary()
summary(ocr)

/Users/wingsangvincentliu/.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/pipeline/tex │
│ t/summary.py:16 in __init__                                                                      │
│                                                                                                  │
│   13 │   """                                                                                     │
│   14 │                                                                                           │
│   15 │   def __init__(self, path=None, quantize=False, gpu=True, model=None, **kwargs):          │
│ ❱ 16 │   │   super().__init__("summarization", path, quantize, gpu, model, **kwargs)             │
│   17 │                                                                                           │
│   18 │   def __call__(self, text, minlength=None, maxlength=None, workers=0):                    │
│   19 │   │   """                                                                                 │
│                                                                                                  │
│ /Users/wingsangvincentliu/.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/pipeline/hfp │
│ ipeline.py:56 in __init__                                                                        │
│                                                                                                  │
│   53 │   │   │   │   self.pipeline = pipeline(task, model=model, tokenizer=path[1], device=de    │
│   54 │   │   │   else:                                                                           │
│   55 │   │   │   │   print(deviceid)                                                             │
│ ❱ 56 │   │   │   │   self.pipeline = pipeline(task, model=path, device=deviceid, model_kwargs    │
│   57 │   │   │                                                                                   │
│   58 │   │   │   # Model quantization. Compresses model to int8 precision, improves runtime p    │
│   59 │   │   │   if deviceid == -1 and quantize:                                                 │
│                                                                                                  │
│ /Users/wingsangvincentliu/.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/pipel │
│ ines/__init__.py:976 in pipeline                                                                 │
│                                                                                                  │
│   973 │   if device is not None:                                                                 │
│   974 │   │   kwargs["device"] = device                                                          │
│   975 │                                                                                          │
│ ❱ 976 │   return pipeline_class(model=model, framework=framework, task=task, **kwargs)           │
│   977                                                                                            │
│                                                                                                  │
│ /Users/wingsangvincentliu/.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/pipel │
│ ines/text2text_generation.py:65 in __init__                                                      │
│                                                                                                  │
│    62 │   return_name = "generated"                                                              │
│    63 │                                                                                          │
│    64 │   def __init__(self, *args, **kwargs):                                                   │
│ ❱  65 │   │   super().__init__(*args, **kwargs)                                                  │
│    66 │   │                                                                                      │
│    67 │   │   self.check_model_type(                                                             │
│    68 │   │   │   TF_MODEL_FOR_SEQ_TO_SEQ_CAUSAL_LM_MAPPING                                      │
│                                                                                                  │
│ /Users/wingsangvincentliu/.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/pipel │
│ ines/base.py:773 in __init__                                                                     │
│                                                                                                  │
│    770 │   │   self.framework = framework                                                        │
│    771 │   │                                                                                     │
│    772 │   │   if self.framework == "pt" and device is not None and not (isinstance(device, int  │
│ ❱  773 │   │   │   self.model.to(device)                                                         │
│    774 │   │                                                                                     │
│    775 │   │   if device is None:                                                                │
│    776 │   │   │   # `accelerate` device map                                                     │
│                                                                                                  │
│ /Users/wingsangvincentliu/.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/model │
│ ing_utils.py:1808 in to                                                                          │
│                                                                                                  │
│   1805 │   │   │   │   " model has already been set to the correct devices and casted to the co  │
│   1806 │   │   │   )                                                                             │
│   1807 │   │   else:                                                                             │
│ ❱ 1808 │   │   │   return super().to(*args, **kwargs)                                            │
│   1809 │                                                                                         │
│   1810 │   def half(self, *args):                                                                │
│   1811 │   │   # Checks if the model has been loaded in 8-bit                                    │
│                                                                                                  │
│ /Users/wingsangvincentliu/.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/m │
│ odule.py:1145 in to                                                                              │
│                                                                                                  │
│   1142 │   │   │   │   │   │   │   non_blocking, memory_format=convert_to_format)                │
│   1143 │   │   │   return t.to(device, dtype if t.is_floating_point() or t.is_complex() else No  │
│   1144 │   │                                                                                     │
│ ❱ 1145 │   │   return self._apply(convert)                                                       │
│   1146 │                                                                                         │
│   1147 │   def register_full_backward_pre_hook(                                                  │
│   1148 │   │   self,                                                                             │
│                                                                                                  │
│ /Users/wingsangvincentliu/.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/m │
│ odule.py:797 in _apply                                                                           │
│                                                                                                  │
│    794 │                                                                                         │
│    795 │   def _apply(self, fn):                                                                 │
│    796 │   │   for module in self.children():                                                    │
│ ❱  797 │   │   │   module._apply(fn)                                                             │
│    798 │   │                                                                                     │
│    799 │   │   def compute_should_use_set_data(tensor, tensor_applied):                          │
│    800 │   │   │   if torch._has_compatible_shallow_copy_type(tensor, tensor_applied):           │
│                                                                                                  │
│ /Users/wingsangvincentliu/.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/m │
│ odule.py:797 in _apply                                                                           │
│                                                                                                  │
│    794 │                                                                                         │
│    795 │   def _apply(self, fn):                                                                 │
│    796 │   │   for module in self.children():                                                    │
│ ❱  797 │   │   │   module._apply(fn)                                                             │
│    798 │   │                                                                                     │
│    799 │   │   def compute_should_use_set_data(tensor, tensor_applied):                          │
│    800 │   │   │   if torch._has_compatible_shallow_copy_type(tensor, tensor_applied):           │
│                                                                                                  │
│ /Users/wingsangvincentliu/.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/m │
│ odule.py:820 in _apply                                                                           │
│                                                                                                  │
│    817 │   │   │   # track autograd history of `param_applied`, so we have to use                │
│    818 │   │   │   # `with torch.no_grad():`                                                     │
│    819 │   │   │   with torch.no_grad():                                                         │
│ ❱  820 │   │   │   │   param_applied = fn(param)                                                 │
│    821 │   │   │   should_use_set_data = compute_should_use_set_data(param, param_applied)       │
│    822 │   │   │   if should_use_set_data:                                                       │
│    823 │   │   │   │   param.data = param_applied                                                │
│                                                                                                  │
│ /Users/wingsangvincentliu/.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/m │
│ odule.py:1143 in convert                                                                         │
│                                                                                                  │
│   1140 │   │   │   if convert_to_format is not None and t.dim() in (4, 5):                       │
│   1141 │   │   │   │   return t.to(device, dtype if t.is_floating_point() or t.is_complex() els  │
│   1142 │   │   │   │   │   │   │   non_blocking, memory_format=convert_to_format)                │
│ ❱ 1143 │   │   │   return t.to(device, dtype if t.is_floating_point() or t.is_complex() else No  │
│   1144 │   │                                                                                     │
│   1145 │   │   return self._apply(convert)                                                       │
│   1146                                                                                           │
│                                                                                                  │
│ /Users/wingsangvincentliu/.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/cuda/__init_ │
│ _.py:239 in _lazy_init                                                                           │
│                                                                                                  │
│    236 │   │   │   │   "Cannot re-initialize CUDA in forked subprocess. To use CUDA with "       │
│    237 │   │   │   │   "multiprocessing, you must use the 'spawn' start method")                 │
│    238 │   │   if not hasattr(torch._C, '_cuda_getDeviceCount'):                                 │
│ ❱  239 │   │   │   raise AssertionError("Torch not compiled with CUDA enabled")                  │
│    240 │   │   if _cudart is None:                                                               │
│    241 │   │   │   raise AssertionError(                                                         │
│    242 │   │   │   │   "libcudart functions unavailable. It looks like you have a broken build?

[Vincent-liuwingsang]

interestingly when I pass in Models.device(deviceId) instead of deviceId works

txtai/src/python/txtai/pipeline/hfpipeline.py

Line 55 in ff2a1c7

self.pipeline = pipeline(task, model=path, device=deviceid, model_kwargs=modelargs, **kwargs)

[davidmezzetti]

I'll see what I can do to debug without access to a mac.

[davidmezzetti]

I was able to reproduce this with the latest version of transformers and GitHub actions. You basically have the fix in your comment above. That is how the other txtai components are building devices.