runs on CPU

[Elfftwelff]

Even though it installed all the cuda packages it still runs on the CPU, does anyone know how to fix this?

[Akshay1-6180]

Yeah even i am facing with this same issue @chaoyi-wu could u write the code to make it run on gpu too

[Akshay1-6180]

To be frank, the number of unresolved issues on the repository is alarming. It's evident that many users, including myself, have invested time and effort trying to integrate your model into our work, only to encounter numerous problems. What exacerbates the situation is the apparent lack of response or action . It's not just about the issues themselves; it's about the seeming disregard for those who have tried to seek guidance or solutions.Anyway here is a code that works with gpu

import transformers
import torch
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = transformers.AutoTokenizer.from_pretrained(
        'axiong/PMC_LLaMA_13B',unk_token='<unk>'
    )
sentence = 'Hello, doctor i have cancer , what treatment should i opt for' 
inputs = tokenizer([sentence])
inputs = {k: torch.tensor(v).to(DEVICE) for k, v in inputs.items()}

model = transformers.AutoModelForCausalLM.from_pretrained(
        'axiong/PMC_LLaMA_13B',
        torch_dtype=torch.bfloat16,
        low_cpu_mem_usage=True,
        device_map="auto",
    )


with torch.no_grad():
    generated = model.generate(
        **inputs,
        max_length=200,
        do_sample=True,
        top_k=50
    )
    print('model predict: ',tokenizer.decode(generated[0]))

[WeixiongLin]

Sorry for the late response, and thanks for your contribution. We are testing our code on colab, and I hope it would help users to find it more accessible. We will let you know as soon as the notebook is ready.

[Akshay1-6180]

thanks for the response , eagerly waiting for the notebook
And btw you guys are doing an amazing job , so kudos to that.