Using CUDA

[ItsMeJacky]

How to run faster-whisper using CUDA ?

[rufinus]

cudnn libs + --device cuda

on my LXC setup i followed this article https://sluijsjes.nl/2024/05/18/coral-and-nvidia-passthrough-for-proxmox-lxc-to-install-frigate-video-surveillance-server/

you just have to include the correct libs in the container, or map damn from the host to the container.

i ended up building the cointainer myself with this dockerfile:

FROM rhasspy/wyoming-whisper:2.1.0
RUN apt update && apt install -y wget software-properties-common
RUN wget https://developer.download.nvidia.com/compute/cudnn/8.9.7.29/local_installers/12.2/cudnn-local-repo-debian11-8.9.7.29_1.0-1_amd64.deb
RUN dpkg -i cudnn-local-repo-debian11-8.9.7.29_1.0-1_amd64.deb
RUN cp /var/cudnn-local-repo-debian11-8.9.7.29/cudnn-*-keyring.gpg /usr/share/keyrings/
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/debian11/x86_64/cuda-keyring_1.1-1_all.deb
RUN dpkg -i cuda-keyring_1.1-1_all.deb
RUN add-apt-repository contrib && apt --allow-releaseinfo-change update
RUN apt -y install libcudnn8 libcublas-12-0
RUN rm cudnn-local-repo-debian11-8.9.7.29_1.0-1_amd64.deb

took some time to find the right versions to make it run.

[ssamjh]

👍 For official GPU support, would be nice to speed things up :)

[ssamjh]

👍 for this feature. I already have a GPU in my server for other applications :)

[Mithras]

Also, is it possible to make it unload model based on some --keep-alive param?

[alienatedsec]

How to run faster-whisper using CUDA ?

Simple and works out of the box - https://github.com/linuxserver/docker-faster-whisper

[AnkushMalaker]

I made PR #44 to allow building it directly from this repository, and use CUDA.
It works perfectly with just docker compose up -d or using the build scripts in the README.
@synesthesiam will need to upload a gpu based image to rhasspy dockerhub if they want to allow people to pull without building locally.

[danieljkemp]

RUN apt update && apt install -y wget software-properties-common \
  && wget https://developer.download.nvidia.com/compute/cudnn/9.1.0/local_installers/cudnn-local-repo-debian11-9.1.0_1.0-1_amd64.deb \
  && dpkg -i cudnn-local-repo-debian11-9.1.0_1.0-1_amd64.deb \
  && cp /var/cudnn-local-repo-debian11-9.1.0/cudnn-*-keyring.gpg /usr/share/keyrings/ \
  && wget https://developer.download.nvidia.com/compute/cuda/repos/debian11/x86_64/cuda-keyring_1.1-1_all.deb \
  && dpkg -i cuda-keyring_1.1-1_all.deb \
  && add-apt-repository contrib && apt --allow-releaseinfo-change update \
  && apt -y install libcudnn9-cuda-12 libcublas-12-0 \
  && rm cudnn-local-repo-debian11-9.1.0_1.0-1_amd64.deb

working with 2.4.0