Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to choose GPU with AllGPT binding. #13

Open
ediweber opened this issue Nov 7, 2023 · 2 comments
Open

How to choose GPU with AllGPT binding. #13

ediweber opened this issue Nov 7, 2023 · 2 comments

Comments

@ediweber
Copy link

ediweber commented Nov 7, 2023

Hello.

First things first: Thank you for creating lollms, it is simply spoken wonderful!

Unfortunately I suffer with the AllGPT binding. I cannot load my GGUF models, output is:

Lollms webui version : 6.8
Listing all extensions
Listing all personalities
Listing models
Loading discussion for client mrQYbMGCR0JO3rJSAAAB
Checking for updates from E:\lollms-webui
update availability: True
Listing all extensions

  • Retreiving binding settings
    Requested updating of setting model_name to Naberius-7B-GGUF
    Changing model to: Naberius-7B-GGUF
    Loading model:naberius-7b.Q8_0.gguf
    Folder:E:\loldata\models\gguf
    Error allocating memory ErrorOutOfDeviceMemory
    error loading model: Error allocating vulkan memory.
    llama_load_model_from_file: failed to load model
    LLAMA ERROR: failed to load model from E:\\loldata\\models\\gguf\naberius-7b.Q8_0.gguf
    Couldn't load model: [Model not loaded]
    Traceback (most recent call last):
    File "E:\lollms-webui\app.py", line 914, in update_setting
    self.model = self.binding.build_model()
    File "E:\lollms-webui\zoos\bindings_zoo\gpt_4all_init_.py", line 120, in build_model
    self.model.model.set_thread_count(self.binding_config.n_threads)
    File "C:\Users\eduar\anaconda3\envs\lol\lib\site-packages\gpt4all\pyllmodel.py", line 274, in set_thread_count
    raise Exception("Model not loaded")
    Exception: Model not loaded

I suspect that it tries to use my APU (Ryzen 5600) @ 4GB instead of my Nvidia 3060 - 12 GB, is there a simple way to force it to use the Nvidia card.

Will try with smaller models. In fact it would be nice to have a small one running on the APU and a large one on the dedicated GPU card.

@ediweber
Copy link
Author

ediweber commented Nov 7, 2023

sorry. I accidently posted this in lollms while this is an lollms-webui issue

@ParisNeo
Copy link
Owner

ParisNeo commented Nov 8, 2023

Well, in the parameters of gpt4all yo ucan select GPU (cuda)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants