About CUDA forward compatibility.

[XunHang]

I experimented with tabby earlier this year and then shut down my own tabby server for some reason. Recently, tabby added support for C++, and I wanted to use it again. However, when I updated the docker image and downloaded the new TabbyML model, the following problem(CUDA 804)occurred. This seems to be related to the version of my system kernel and the version of the NVIDIA graphics card. , I don’t dare to update easily, so I would like to ask everyone, what is the correct way to do it.

我在今年早些时候，体验了 tabby，之后由于一些原因关闭了我自己的 tabby server。最近，tabby 增加了对 C++ 的支持，我又想来使用一下了，但是当我更新了镜像并下载了模型后，出现了下面的 CUDA 804 问题，这似乎和我的系统内核版本与英伟达显卡版本有关，我不敢轻易的进行更新，因此想来请教一下大家，怎么做才是正确的办法。

This is the tabby error.

[~/.tabby/models/TabbyML/DeepseekCoder-6.7B/ggml] 
❱❱❱ docker run -it --gpus all -p 8080:8080 -v $HOME/.tabby:/data tabbyml/tabby serve --model TabbyML/DeepseekCoder-6.7B --device cuda
2023-12-25T07:12:14.372199Z  INFO tabby::serve: crates/tabby/src/serve.rs:111: Starting server, this might takes a few minutes...
2023-12-25T07:12:14.376378Z  INFO tabby::services::code: crates/tabby/src/services/code.rs:53: Index is ready, enabling server...    

CUDA error 804 at /root/workspace/crates/llama-cpp-bindings/llama.cpp/ggml-cuda.cu:478: forward compatibility was attempted on non supported HW
current device: 0

This is my nvidia-smi information.

nvidia-smi  
Tue Dec 26 08:54:57 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01    Driver Version: 470.63.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 32%   25C    P3    28W / 170W |    655MiB / 12031MiB |     10%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1116      G   /usr/lib/xorg/Xorg                366MiB |
|    0   N/A  N/A      2101      G   /usr/bin/gnome-shell              105MiB |
|    0   N/A  N/A      5327      G   /proc/self/exe                     22MiB |
|    0   N/A  N/A    161513      G   ...RendererForSitePerProcess        9MiB |
|    0   N/A  N/A   2017255      G   ...eatures=msEdgeSplitWindow      127MiB |
|    0   N/A  N/A   2493820      G   ...local/kitty.app/bin/kitty       14MiB |
+-----------------------------------------------------------------------------+

And this is my cuda and cudnn libraries

[/usr/lib/x86_64-linux-gnu] 
❱❱❱ ls | grep -e "cuda\|cudnn"
libcuda.so
libcuda.so.1
libcuda.so.470.63.01
libcudnn_adv_infer.so
libcudnn_adv_infer.so.8
libcudnn_adv_infer.so.8.2.6
libcudnn_adv_train.so
libcudnn_adv_train.so.8
libcudnn_adv_train.so.8.2.6
libcudnn_cnn_infer.so
libcudnn_cnn_infer.so.8
libcudnn_cnn_infer.so.8.2.6
libcudnn_cnn_infer_static.a
libcudnn_cnn_infer_static_v8.a
libcudnn_cnn_train.so
libcudnn_cnn_train.so.8
libcudnn_cnn_train.so.8.2.6
libcudnn_cnn_train_static.a
libcudnn_cnn_train_static_v8.a
libcudnn_ops_infer.so
libcudnn_ops_infer.so.8
libcudnn_ops_infer.so.8.2.6
libcudnn_ops_train.so
libcudnn_ops_train.so.8
libcudnn_ops_train.so.8.2.6
libcudnn.so
libcudnn.so.8
libcudnn.so.8.2.6
libcudnn_static.a
libcudnn_static_v8.a
libicudata.a
libicudata.so
libicudata.so.66
libicudata.so.66.1

[XunHang]

Please add some additional information about my device configuration.

GPU: NVIDIA GeForce RTX 3060
OS: Ubuntu 20.04.4 LTS
Linux CORE: 5.13.0-30-generic

[wsxiaoys]

NVIDIA-SMI 470.63.01 Driver Version: 470.63.01 CUDA Version: 11.4

Tabby 0.7.0 is compiled on cuda 11.7 - could you try upgrading your cuda version?

[XunHang]

Thank you for giving the specific version!
I will try to upgrade later to solve the cuda problem caused by llama.cpp.

In addition, if there is an upgrade to CUDA in the future, please note it in the release note, which maybe help users.

Thanks again for your reply (╹▽╹)

[XunHang]

Sorry, I just learned that cuda generally needs to be used with cudnn. Do I need to update cudnn to adapt to the 11.7 version of cuda?

抱歉，我刚刚了解到 cuda 一般需要搭配 cudnn 来使用，我是否需要更新 cudnn 来与 11.7 版本的 cuda 进行适配？ (O_O)?