CUDA not found

[jlb1504]

Hey, this is really great, and I have got it working. I have a gaming laptop with 2 GPUs: an AMD with 512 MB VRAM, and a RTX 4080 with 12 GB VRAM. The model is running pretty well, but it is hitting the AMD, not the RTX. When launched, I am getting "CUDA not found". I am doing all this on Win11/WSL2. I think I have cuda installed and enabled properly for python in WSL:

torch.cuda.is_available()
True

Here is my .env:

MODEL_PATH = "TheBloke/Llama-2-7b-Chat-GPTQ"
LOAD_IN_8BIT = False
LOAD_IN_4BIT = True
LLAMA_CPP = False

MAX_MAX_NEW_TOKENS = 2048
DEFAULT_MAX_NEW_TOKENS = 1024
MAX_INPUT_TOKEN_LENGTH = 4000

(Note: I had to tweak MODEL_PATH from example - this value above worked for me. I also removed DEFAULT_SYSTEM_PROMPT for now, because it was crashing the app with "everett.ConfigurationError: Env file line missing = operator")

Do you have any suggestions on what I might need to do to get the app to recognize and use the RTX? Thanks!

[jlb1504]

It does seem like it briefly hits the RTX. I am pretty new to a lot of this stuff, so I may not understand what I am looking at.

[liltom-eth]

@jlb1504 Thanks for the issue. That must be a bug here. not should not be in that line.
That's my fault.

[liltom-eth]

Bug should be fixed with the newest commits now.

[jlb1504]

ok, so it was/is using cuda - it was just printing out the opposite. I now see this message on launch: "Running on GPU with torch transformers." Thanks!

[liltom-eth]

@jlb1504 Thanks for contributing, leave your Twitter name here if you like to be mentioned in some posts later.

[liltom-eth]

@jlb1504 welcome contributing your benchmark performance here.