Add nvidia libraries for hardware-accelerated en-/decoding

[psych0d0g]

Please add librarys required for HW acceleration to work to the container.

Apr 07, 2021 12:12:08.892 [0x7fd516ffd640] ERROR - [Transcode] [FFMPEG] - Cannot load libnvidia-encode.so.1
Apr 07, 2021 12:12:08.892 [0x7fd516ffd640] ERROR - [Transcode] [FFMPEG] - The minimum required Nvidia driver for nvenc is 418.30 or newer
Apr 07, 2021 12:12:08.934 [0x7fd516ffd640] WARN - [Transcode] Codecs: avcodec_open2 returned -1 for encoder 'h264_nvenc'
Apr 07, 2021 12:12:09.164 [0x7fd516ffd640] ERROR - [Transcode] [FFMPEG] - Cannot load libnvidia-encode.so.1
Apr 07, 2021 12:12:09.164 [0x7fd516ffd640] ERROR - [Transcode] [FFMPEG] - The minimum required Nvidia driver for nvenc is 418.30 or newer
Apr 07, 2021 12:12:09.200 [0x7fd516ffd640] WARN - [Transcode] Codecs: avcodec_open2 returned -1 for encoder 'h264_nvenc'
Apr 07, 2021 12:12:09.990 [0x7fd532515640] ERROR - Error issuing curl_easy_perform(handle): 3
Apr 07, 2021 12:12:09.990 [0x7fd532515640] WARN - HTTP error requesting GET /media/providers (3, Error) ()
Apr 07, 2021 12:12:10.188 [0x7fd532515640] ERROR - Unknown metadata type: 
Apr 07, 2021 12:12:11.403 [0x7fd4ff7fe640] ERROR - [Transcoder] [h264 @ 0x393a700] Cannot load libnvcuvid.so.1
Apr 07, 2021 12:12:11.403 [0x7fd4bbfff640] ERROR - [Transcoder] [h264 @ 0x393a700] Failed loading nvcuvid.
Apr 07, 2021 12:12:11.404 [0x7fd516ffd640] ERROR - [Transcoder] [h264 @ 0x393a700] Failed setup for format cuda: hwaccel initialisation returned error.

[psych0d0g]

not sure how one would add this properly to the container image,
i was able to get my local plex to transcode using nvenc hardware by copying the following files from the host into the container FS:

libnvcuvid.so
libnvidia-allocator.so.460.67
libnvidia-compiler.so.460.67
libnvidia-encode.so
libnvidia-fbc.so.460.67
libnvidia-ifr.so
libnvidia-ml.so.460.67
libnvidia-opticalflow.so
libnvidia-ptxjitcompiler.so.460.39
libnvcuvid.so.1
libnvidia-cbl.so.460.67
libnvidia-container.so.1
libnvidia-encode.so.1
libnvidia-glcore.so.460.67
libnvidia-ifr.so.1
libnvidia-ngx.so.1
libnvidia-opticalflow.so.1
libnvidia-ptxjitcompiler.so.460.67
libnvcuvid.so.460.39
libnvidia-cfg.so
libnvidia-container.so.1.3.3
libnvidia-encode.so.460.39
libnvidia-glsi.so.460.67
libnvidia-ifr.so.460.67
libnvidia-ngx.so.460.67
libnvidia-opticalflow.so.460.39
libnvidia-rtcore.so.460.67
libnvcuvid.so.460.67
libnvidia-cfg.so.1
libnvidia-egl-wayland.so.1
libnvidia-encode.so.460.67
libnvidia-glvkspirv.so.460.67
libnvidia-ml.so
libnvidia-opencl.so.1
libnvidia-opticalflow.so.460.67
libnvidia-tls.so.460.67
libnvidia-allocator.so
libnvidia-cfg.so.460.67
libnvidia-egl-wayland.so.1.1.5
libnvidia-fbc.so
libnvidia-gtk2.so.460.67
libnvidia-ml.so.1
libnvidia-opencl.so.460.39
libnvidia-ptxjitcompiler.so
libnvoptix.so.1
libnvidia-allocator.so.1
libnvidia-compiler.so.460.39
libnvidia-eglcore.so.460.67
libnvidia-fbc.so.1
libnvidia-gtk3.so.460.67
libnvidia-ml.so.460.39
libnvidia-opencl.so.460.67
libnvidia-ptxjitcompiler.so.1
libnvoptix.so.460.67

[frebib]

I could create a branch that builds onto the base image and adds in those libraries. I'm reluctant to add them to the master/pass branch images though purely due to size. Can I ask what is the rough size of those library files in total?
Also which tag of this image do you use?
I'll see what I can pull together. For now if you wanted a quick fix, something like this (completely untested btw) should get you by until we manage to work out a more permanent solution:

FROM debian:bullseye
RUN apt-get update -y && apt-get install -y <whatever-nvidia-package-has-those-libs>
FROM spritsail/plex-media-server:<tag>
COPY --from=0 /usr/lib/libnvidia*.so* /usr/lib/nvcu*.so* /usr/lib/nvoptix.so* /usr/lib

[psych0d0g]

if i can help any further i would be happy to.
Sadly, the libs are friggin humoungous (~500mb), ill try and check if i really need the whole collection of nvidia libs and if i can remove some and still have HW transcoding working properly

[frebib]

It's likely that most of them aren't required. I see two listed in the logs you provided. They might have a few dependencies which you can see with ldd <path/to/lib.so>. Be sure to post your results here and let me know how it goes.
It's probably a good idea to start with those two, and any immediate dependencies they may have. Any others can be added iteratively as you find they're missing

[psych0d0g]

libnvoptix.so.460.67 can surely be removed from the list, trims down overall space by 220mb already

[psych0d0g]

ldd libnvidia-encode.so.1
	linux-vdso.so.1 (0x00007ffd48571000)
	libnvcuvid.so.1 => /usr/lib/libnvcuvid.so.1 (0x00007f0491c94000)
	libc.so.6 => /lib/libc.so.6 (0x00007f0491b0f000)
	libdl.so.2 => /lib/libdl.so.2 (0x00007f0491b0a000)
	libm.so.6 => /lib/libm.so.6 (0x00007f04919e3000)
	/usr/lib/ld-linux-x86-64.so.2 (0x00007f049257c000)

ldd libnvcuvid.so.1
	linux-vdso.so.1 (0x00007ffda6ed9000)
	libm.so.6 => /lib/libm.so.6 (0x00007f13b724e000)
	libc.so.6 => /lib/libc.so.6 (0x00007f13b70c9000)
	libdl.so.2 => /lib/libdl.so.2 (0x00007f13b70c4000)
	/usr/lib/ld-linux-x86-64.so.2 (0x00007f13b7a41000)

so it "seems" that we just need those 2 libs, just checked and plex sucessfully transcodes using HW acceleration with only those 2 files

sizewize:

cronix@mediasrv:/mnt/drives/libs$ ls -lah libnvcuvid.so
-rwxr-xr-x 1 root root 4.8M Apr  7 12:48 libnvcuvid.so
cronix@mediasrv:/mnt/drives/libs$ ls -lah libnvidia-encode.so
-rwxr-xr-x 1 root root 111K Apr  7 12:51 libnvidia-encode.so

[psych0d0g]

from ubuntu packages:

cronix@mediasrv:/mnt/drives/libs$ dpkg -L libnvidia-encode-460
/.
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/libnvidia-encode.so.460.39
/usr/share
/usr/share/doc
/usr/share/doc/libnvidia-encode-460
/usr/share/doc/libnvidia-encode-460/changelog.Debian.gz
/usr/share/doc/libnvidia-encode-460/copyright
/usr/lib/x86_64-linux-gnu/libnvidia-encode.so
/usr/lib/x86_64-linux-gnu/libnvidia-encode.so.1

and the cuvid lib is in the corresponding decode package:

cronix@mediasrv:/mnt/drives/libs$ dpkg -L libnvidia-decode-460
/.
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/libnvcuvid.so.460.39
/usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.460.39
/usr/lib/x86_64-linux-gnu/vdpau
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_nvidia.so.460.39
/usr/share
/usr/share/doc
/usr/share/doc/libnvidia-decode-460
/usr/share/doc/libnvidia-decode-460/changelog.Debian.gz
/usr/share/doc/libnvidia-decode-460/copyright
/usr/lib/x86_64-linux-gnu/libnvcuvid.so
/usr/lib/x86_64-linux-gnu/libnvcuvid.so.1
/usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so
/usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.1
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_nvidia.so
/usr/lib/x86_64-linux-gnu/vdpau/libvdpau_nvidia.so.1

[frebib]

Great investigation! Thanks. I'll put something together later

[psych0d0g]

thanks so much, great work on all these spiritsail containers btw 👍
do you guys have some kind of chat for exchange?

[frebib]

I've pushed some images to Docker Hub if you want to give them a try. The source for that image is on the nvidia branch.
https://hub.docker.com/r/spritsail/plex-media-server/tags?page=1&ordering=last_updated&name=nvidia

Let me know if it works or not. I don't have Docker set up with nvidia yet. Last I saw it was a bit of a pain to try to get working. I might try setting it up if this image doesn't work for you.

As for chat, I don't think we do really. You can always open a GitHub issue for problems (I'd always prefer these for tracking). You can email us @spritsail.io, the email addresses should be in the maintainer labels for each of the images. I guess you could always hit me up on Matrix @frebib:chat.nerdhouse.io or Telegram @frebib. I'll have a think about the chat thing though, I think it could be a good idea

[frebib]

Bump @psych0d0g

[psych0d0g]

ill check the build soon, sorry had a bit of private troubles lately that held me back from experimenting

[lwndow]

I stumbled across this tagged image looking for something similar. Would you be able to bump the pass version with these libs @frebib and I'll be another tester?

[frebib]

@lwndow I'm 99.9% sure that this is broken since Plex moved to musl as it's libc, but I've rebased the nvidia branch and re-pushed it but I think CI will likely fail: https://drone.spritsail.io/spritsail/plex-media-server/695

I'm not sure how to make this work with musl as I expect nvidia don't provide anything other than the glibc-based libraries that Debian has. I briefly looked for source/upstream for these a while back but couldn't find it. Presumably this does still work with other images though so it'll require a little more work on my part

Edit: I made it build, but it won't work, at least with the Debian-packaged libraries

/usr/lib # /lib/ld-musl-x86_64.so.1 --list libnvcuvid.so.465.31
	/lib/ld-musl-x86_64.so.1 (0x7fbfaddf4000)
	libm.so.6 => /lib/ld-musl-x86_64.so.1 (0x7fbfaddf4000)
	libc.so.6 => /lib/ld-musl-x86_64.so.1 (0x7fbfaddf4000)
	libdl.so.2 => /lib/ld-musl-x86_64.so.1 (0x7fbfaddf4000)
Error relocating libnvcuvid.so.465.31: dlvsym: symbol not found
/usr/lib # /lib/ld-musl-x86_64.so.1 --list libnvidia-encode.so.465.31
	/lib/ld-musl-x86_64.so.1 (0x7fd21b383000)
Error loading shared library libnvcuvid.so.1: No such file or directory (needed by libnvidia-encode.so.465.31)
	libc.so.6 => /lib/ld-musl-x86_64.so.1 (0x7fd21b383000)
	libdl.so.2 => /lib/ld-musl-x86_64.so.1 (0x7fd21b383000)
Error relocating libnvidia-encode.so.465.31: dlvsym: symbol not found

[frebib]

NVIDIA/nvidia-docker#509 (comment)

Time to buy an AMD card I think 😁

[lwndow]

NVIDIA/nvidia-docker#509 (comment)

Time to buy an AMD card I think 😁

DEFINITELY

[Drikani]

Is it at all possible to use a nvidia card with this container or do I need to go for the official pms-docker or linuxserver image?
I use it on UnRAID and plan on getting a used GTX 1060 for transcoding with tdarr as well and everywhere I look it only tells me to add a few variables and runtime options and the rest will be done by the plugin in UnRAID?
Does this Issue here only apply to those who do not use the nvidia-docker container runtime?

[frebib]

There's not much we can do here until nvidia either provide the source for their libraries (never going to happen) or they provide binaries compiled against musl instead of glibc. We could also bundle glibc into our image with these libraries which I think should work, but that's a bit of work to do. I'll accept a PR if anyone wants to give it a go.