aten::index_copy.out

[vbharath18]

NotImplementedError: The operator 'aten::index_copy.out' is not currently implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on pytorch/pytorch#77764. As a temporary fix, you can set the environment variable PYTORCH_ENABLE_MPS_FALLBACK=1 to use the CPU as a fallback for this op. WARNING: this will be slower than running natively on MPS.

Getting this in 05_part-2.ipynb.

I use MacBook M3 Pro 18GB RAM.

[rasbt]

Hi there, thanks for sharing. I don't remember creating a 05_part-2.ipynb notebook. Do you mean ch05.ipynb perhaps? Although this should work on MPS devices.

[vbharath18]

Apologies, this is from the repo - https://github.com/rasbt/LLM-workshop-2024/tree/main/05_weightloading

[rasbt]

No worries and thanks! Yes, I can confirm this issue on MacBooks. If you change the code

from litgpt import LLM

llm = LLM.load("microsoft/phi-2")
llm.generate("What do Llamas eat?")

to

from litgpt import LLM

llm = LLM.load("microsoft/phi-2")

llm.distribute(accelerator="cpu")
llm.generate("What do Llamas eat?")

it should work.

I will look into how to adjust that in LitGPT that we use an alternative to the index_copy_ op in LitGPT if MPS devices are used.

[vbharath18]

Tried changing this, I'm getting a runtime error now.

`RuntimeError Traceback (most recent call last)
Cell In[2], line 6
3 llm = LLM.load("microsoft/phi-2")
5 llm.distribute(accelerator="cpu")
----> 6 llm.generate("What do Llamas eat?")

File ~/miniforge3/envs/LLMs/lib/python3.10/site-packages/torch/utils/_contextlib.py:116, in context_decorator..decorate_context(*args, **kwargs)
113 @functools.wraps(func)
114 def decorate_context(*args, **kwargs):
115 with ctx_factory():
--> 116 return func(*args, **kwargs)

File ~/miniforge3/envs/LLMs/lib/python3.10/site-packages/litgpt/api.py:534, in LLM.generate(self, prompt, max_new_tokens, temperature, top_k, top_p, return_as_token_ids, stream)
532 outputs = iterator()
533 else:
--> 534 outputs = generate_fn(
535 model=self.model,
536 prompt=input_ids,
537 max_returned_tokens=max_returned_tokens,
538 temperature=temperature,
539 top_k=top_k,
540 top_p=top_p,
541 eos_id=self.preprocessor.tokenizer.eos_id,
542 include_prompt=False,
543 )
545 if stream:
546 return outputs

File ~/miniforge3/envs/LLMs/lib/python3.10/site-packages/torch/utils/_contextlib.py:116, in context_decorator..decorate_context(*args, **kwargs)
113 @functools.wraps(func)
114 def decorate_context(*args, **kwargs):
115 with ctx_factory():
--> 116 return func(*args, **kwargs)

File ~/miniforge3/envs/LLMs/lib/python3.10/site-packages/litgpt/generate/base.py:202, in generate(model, prompt, max_returned_tokens, temperature, top_k, top_p, eos_id, include_prompt)
162 @torch.inference_mode()
163 def generate(
164 model: GPT,
(...)
172 include_prompt: bool = True,
173 ) -> torch.Tensor:
174 """
175 Takes a conditioning sequence (prompt) as input and continues to generate as many tokens as requested.
176 The implementation of this function is modified from A. Karpathy's nanoGPT.
(...)
199 include_prompt: If true (default) prepends the prompt (after applying the prompt style) to the output.
200 """
--> 202 token_list = list(generate_fn(
203 include_prompt=include_prompt,
204 include_eos=True,
205 model=model,
206 prompt=prompt,
207 max_returned_tokens=max_returned_tokens,
208 temperature=temperature,
209 top_k=top_k,
210 top_p=top_p,
211 stop_tokens=(([eos_id],) if eos_id is not None else ())
212 ))
214 return torch.cat(token_list) if not len(token_list) == 0 else torch.Tensor()

File ~/miniforge3/envs/LLMs/lib/python3.10/site-packages/torch/utils/_contextlib.py:36, in _wrap_generator..generator_context(*args, **kwargs)
33 try:
34 # Issuing None to a generator fires it up
35 with ctx_factory():
---> 36 response = gen.send(None)
38 while True:
39 try:
40 # Forward the response to our caller and get its next request

File ~/miniforge3/envs/LLMs/lib/python3.10/site-packages/litgpt/generate/base.py:120, in generate_fn(model, prompt, max_returned_tokens, temperature, top_k, top_p, stop_tokens, include_prompt, include_eos)
116 input_pos = torch.arange(0, prompt_size, device=device, dtype=torch.int64)
117 for current_idx in range(max_returned_tokens - prompt_size):
118
119 # Generate the token
--> 120 token = next_token(model, input_pos, token.view(1, -1), temperature=temperature, top_k=top_k, top_p=top_p)
121 tokens.append(token)
122 int_token = token.item()

File ~/miniforge3/envs/LLMs/lib/python3.10/site-packages/litgpt/generate/base.py:78, in next_token(model, input_pos, x, **kwargs)
76 def next_token(model: GPT, input_pos: torch.Tensor, x: torch.Tensor, **kwargs: Any) -> torch.Tensor:
77 logits = model(x, input_pos)
---> 78 next = sample(logits, **kwargs)
79 return next.to(dtype=x.dtype)

File ~/miniforge3/envs/LLMs/lib/python3.10/site-packages/litgpt/generate/base.py:72, in sample(logits, temperature, top_k, top_p)
70 logits = sample_top_p(logits, top_p)
71 probs = torch.nn.functional.softmax(logits, dim=-1)
---> 72 return multinomial_num_samples_1(probs)
73 return torch.argmax(logits, dim=-1, keepdim=True)

File ~/miniforge3/envs/LLMs/lib/python3.10/site-packages/litgpt/generate/base.py:35, in multinomial_num_samples_1(probs)
33 distribution = torch.empty_like(probs).exponential_(1)
34 return torch.argmax(probs / distribution, dim=-1, keepdim=True)
---> 35 return torch.multinomial(probs, num_samples=1)

RuntimeError: probability tensor contains either inf, nan or element < 0`

[rasbt]

Sorry about that, you are right, there something super weird happening when running this on the CPU at the moment. I am currently out of office but will look into it next week.

In the meantime, it works when downgrading to the previous version:

pip install litgpt==0.4.9 -U

[rasbt]

As of the most recent release (v0.4.12), litgpt should work again on macOS. You can either install the latest version from main or do

pip install litgpt -U

Then, e.g.,

from litgpt import LLM

llm = LLM.load("microsoft/phi-2")
llm.generate("What do Llamas eat?")

will produce reasonable results again. (There was an issue with the default precision on macOS that I just fixed.)