Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CUDA backend #2310

Merged
merged 52 commits into from
May 15, 2024
Merged

CUDA backend #2310

merged 52 commits into from
May 15, 2024

Conversation

cebtenzzre
Copy link
Member

@cebtenzzre cebtenzzre commented May 6, 2024 •
edited
Loading

This PR adds opt-in CUDA support in the GPT4All UI and python bindings using the llama.cpp CUDA backend.

CUDA-enabled devices will appear as e.g. "CUDA: Tesla P40" on supported platforms, alongside their Vulkan counterparts. When one is selected, the CUDA backend will be used instead of the Kompute backend.

When CUDA is not available (e.g. a compatible driver or GPU is not installed), CUDA devices simply do not appear. OOM will cause fallback to CPU, just like with Kompute.

The CUDA runtime libraries are installed to the lib/ directory on Linux and Windows (cudart and cublas). Care is taken to make sure the driver component of CUDA, libnvcuda.dll/libcuda.so, is not installed, as it should come with graphics driver.

Other changes:

  • Scaffolding for the Vulkan and ROCm backends (both have serious limitations atm)
  • llama.cpp updated to commit ggerganov/llama.cpp@83330d8cd from May 8th
  • Device selection is now meaningful on macOS: "Metal" (new option) and "Auto" do the same thing, "CPU" now uses the CPU backend instead of Metal

While I was working on packaging the CUDA libraries, I cleaned up a lot of junk that was installed but not needed:

  • DLL import libraries on Windows
  • Static libraries, including Kompute and fmtlib
  • Various headers and cmake scripts from Kompute, Vulkan itself, and fmtlib
  • An extra copy of the llmodel lib (in bin/ on Linux, in lib/ on Windows)

@cebtenzzre
remove lingering references to SOM license
dc03f81 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
backend: bring some upstream changes into llama.cpp.cmake
c482bae 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
backend: update llama.cpp and list of supported models
1684f7b 
Kompute is no longer working.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
backend: bring more upstream changes into llama.cpp.cmake
dc8cd8c 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
llamamodel: remove dependency on internal llama_token_to_piece
75d05c5 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
backend: initial port to Occam's Vulkan backend
1ebd3cf 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
vulkan: improve the implementation
9f92731 
General:
- Proper implementation of gpuDeviceName()
- Make usingGPUDevice() consistent with Kompute impl
- Disable multi-GPU when selecting a specific device (currently: always)

For the bindings:
- Abort instead of segfaulting if multiple LLMs are loaded
- Implement GPU device selection by name/vendor

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
backend: also build CUDA backend by default
ce2164e 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
backend: make CUDA build useful
011935e 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
backend: add option to build ROCm backend
9e457bf 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
cuda: implement device enumeration
c47900e 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
cmake: don't build CPU variant on Linux/Windows
fef041c 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
llmodel: add a backend field to LLModel::GPUDevice
bbe5cc0 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
llmodel: select a backend, not a build variant
dc334ae 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
llmodel: list GPU devices from all backends
b54151f 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
python: implement selectable GPU backend
d4feaeb 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
chat: implement basic UI backend selection
1a10587 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
cmake: set RUNPATH of llamamodel-mainline-cuda correctly
a9ffe5a 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
backend: fix kompute-avxonly build
d30d5b2 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
cuda: fix dependency bundling on Windows
16170f4 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
ci: install CUDA toolkit
2417105 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
cuda: ignore libcuda.so.* on Linux
9e8f7c3 
This file is part of the graphics driver and should not be bundled with
GPT4All.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
ci: install additional deps to make linuxdeployqt happy
9e54277 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
cmake: do not ship static llama library
335b129 
llama.cpp itself is unconditionally built as a static library.
Installing it with the GUI is pointless.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
llama.cpp: do not install kompute or fmt
b14992c 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
cmake: let linuxdeployqt handle CUDA deps on Linux
2b5d6d3 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre cebtenzzre linked an issue May 10, 2024 that may be closed by this pull request
[Feature] Add support for StarCoder2 model architecture #2100
Closed
@cebtenzzre
build_and_run: mention compiler and CUDA
21bc8c8 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
settings: prefix vk devices with "Vulkan: ", update old names
0cbca27 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
kompute: fix device name leaks
8dbe93d 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre cebtenzzre self-assigned this May 15, 2024
@manyoso
Copy link
Collaborator

manyoso commented May 15, 2024

Can you create an offline installer with this change for linux for testing please?

manyoso
manyoso requested changes May 15, 2024
View reviewed changes
gpt4all-backend/llamamodel.cpp Show resolved Hide resolved
gpt4all-backend/llamamodel.cpp Show resolved Hide resolved
gpt4all-chat/chatllm.cpp Show resolved Hide resolved
@cebtenzzre
chat: bump version to 2.8.0
dbf38b2 
This will be a big release, so increment the minor version.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
Merge branch 'main' into add-cuda-support
1c09b6d
@cebtenzzre
python: update README to reflect CUDA build dependency
c7f8c93 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
python: bump version for CUDA support
5875d83 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre cebtenzzre merged commit d2a99d9 into main May 15, 2024
6 of 19 checks passed
@cebtenzzre cebtenzzre deleted the add-cuda-support branch May 15, 2024 19:27
cebtenzzre added a commit that referenced this pull request May 15, 2024
@cebtenzzre
cmake: fix Metal build after #2310
7c90f2a 
I don't understand why this is needed, but it works.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
cebtenzzre added a commit that referenced this pull request May 15, 2024
@cebtenzzre
cmake: fix Metal build after #2310 (#2350)
bf9aaab 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
manyoso pushed a commit that referenced this pull request May 15, 2024
@cebtenzzre
cmake: fix Metal build after #2310 (#2350)
a92d266 
I don't understand why this is needed, but it works.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
cebtenzzre added a commit that referenced this pull request May 15, 2024
@cebtenzzre
cmake: fix Metal build after #2310 (#2350)
7fe0639 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
cebtenzzre added a commit that referenced this pull request May 21, 2024
@cebtenzzre
ci: explicitly use -DCMAKE_BUILD_TYPE=Release
a275447 
This matters now that #2310 removed the default of "Release" in
llama.cpp.cmake.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre cebtenzzre mentioned this pull request May 21, 2024
Merged
@iimez iimez mentioned this pull request May 26, 2024
Merged
@cebtenzzre cebtenzzre mentioned this pull request May 28, 2024
Merged
cebtenzzre added a commit that referenced this pull request May 28, 2024
@cebtenzzre
llamamodel: fix embedding crash for >512 tokens after #2310
9681f67 
n_ubatch defaults to 512, but as of the latest llama.cpp you cannot pass
more than n_ubatch tokens to the embedding model without hitting an
assertion failure.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre cebtenzzre mentioned this pull request May 28, 2024
Merged
cebtenzzre added a commit that referenced this pull request May 29, 2024
@cebtenzzre
llamamodel: fix embedding crash for >512 tokens after #2310 (#2383)
e94177e 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
This was referenced Jun 4, 2024
Closed
Open
@cebtenzzre cebtenzzre mentioned this pull request Jun 13, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@manyoso manyoso manyoso approved these changes

Assignees

@cebtenzzre cebtenzzre

Labels
None yet
Projects
No open projects
(Archived) GPT4All 2024 Roadmap and Active Issues
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature] Add support for StarCoder2 model architecture
2 participants
@cebtenzzre @manyoso