Hardware Acceleration Unavailable After Update

[krakow10]

Hi, I need help figuring out why hardware acceleration stopped working. I updated from 0.17.1 to 0.17.3 using this procedure:

cd OvenMediaEngine/
git fetch
git checkout 7aae7c1a07caaa6d8ffdc07266ce85988d353047
sudo rm -rf /opt/ovenmediaengine/
misc/prerequisites.sh -o="Ubuntu" --enable-nvc
cd src
make -j16 release
sudo make install
sudo systemctl daemon-reload 
sudo systemctl restart ovenmediaengine.service 
sudo journalctl -u ovenmediaengine.service -e

After this, the service log reports the following:

[OvenMediaEngine:235397] Transcoder | transcoder_gpu.cpp:54   | No supported NVIDIA accelerator

I did not restart the container or the server, it just stopped working with the update.
The nvtop utility is able to report GPU utilization, meaning the nvidia driver is working.

I tried the following versions:

• 0.17.3
• 0.17.1 (went back)
• master

I am using Debian 12 within an LXC on Proxmox VE

Here is my Server.xml file:
Server.xml.log

Here is the ovenmediaengine log file:
ovenmediaengine.log

[naanlizard]

To be clear, did you update the code inside the container? Surely the correct way to update a docker service is to pull the new image

…

On Fri, Dec 20, 2024 at 18:04 krakow10 ***@***.***> wrote: Hi, I need help figuring out why hardware acceleration stopped working. I updated from 0.17.1 to 0.17.3 using this procedure: cd OvenMediaEngine/ git fetch git checkout 7aae7c1 sudo rm -rf /opt/ovenmediaengine/ misc/prerequisites.sh -o="Ubuntu" --enable-nvc cd src make -j16 release sudo make install sudo systemctl daemon-reload sudo systemctl restart ovenmediaengine.service sudo journalctl -u ovenmediaengine.service -e After this, the service log reports the following: [OvenMediaEngine:235397] Transcoder | transcoder_gpu.cpp:54 | No supported NVIDIA accelerator I did not restart the container or the server, it just stopped working with the update. The nvtop utility is able to report GPU utilization, meaning the nvidia driver is working. I tried the following versions: - 0.17.3 - 0.17.1 (went back) - master I am using Debian 12 within an LXC on Proxmox VE Here is my Server.xml file: Server.xml.log <https://github.com/user-attachments/files/18216015/Server.xml.log> Here is the ovenmediaengine log file: ovenmediaengine.log <https://github.com/user-attachments/files/18216018/ovenmediaengine.log> — Reply to this email directly, view it on GitHub <#1755>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABV4BWKZAVS3ES747KPWPHD2GSO7XAVCNFSM6AAAAABT77FABWVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZXG4ZDENRSHE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[krakow10]

There is no docker involved, this is building from source in an LXC

[naanlizard]

Ah, you said container which made me assume. I didn't realize lxc was a full virtualization environment Thank you for clarifying

…

On Fri, Dec 20, 2024 at 18:06 krakow10 ***@***.***> wrote: There is no docker involved, this is building from source in an LXC — Reply to this email directly, view it on GitHub <#1755 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABV4BWP2WYVWLQVN4LFWD5L2GSPIPAVCNFSM6AAAAABT77FABWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNRTGQZDKNA> . You are receiving this because you commented.Message ID: ***@***.*** com>

[krakow10]

I've been fussing with docker for hours with nothing to show for it. Proxmox and docker don't really mix well as far as I can tell. I have an arch linux container on the server as well, so I looked into that since arch linux has ovenmediaengine packaged as version 0.17.2, and it has the same issue where it can't see the nvidia gpu but ffmpeg, nvtop, nvidia-smi all can.

[getroot]

@Keukhan Please check this

[Keukhan]

@krakow10

Can you share the /lib/systemd/system/ovenmediaengine.service file?

Recently, if there have been any changes, the execution has been modified to run the ome_launcher.sh script instead of the OvenMediaEngine file. The ome_launcher script is responsible for locating the path of the installed NVIDIA driver and preloading it.

If the ome_launcher.sh script is being executed, it is assumed that the operating system is not supported.

ome_launcher.sh

check_nvidia_driver() {
    LIB_PATH=""
    LIB_FILES=("libnvidia-ml.so.1" "libcuda.so.1" "libnppicc.so.10" "libnppig.so.10" "libnppicc.so.11" "libnppig.so.11" "libnppicc.so.12" "libnppig.so.12")

    if [ "${OSNAME}" == "Ubuntu" ]; then
        LIB_PATH=/lib/x86_64-linux-gnu
    elif  [ "${OSNAME}" == "CentOS" ] || [ "${OSNAME}" == "Amazon Linux" ] || [ "${OSNAME}" == "Rocky" ]; then
        LIB_PATH=/usr/lib64
    else
        return
    fi

    for LIB_FILE in ${LIB_FILES[@]}; do
        if [ -f ${LIB_PATH}/${LIB_FILE} ]; then
            export LD_PRELOAD=${LIB_PATH}/${LIB_FILE}:${LD_PRELOAD}
        fi
    done
}

It seems the code needs to be modified to ensure it works on Debian 12.

If the driver loading is successful on that OS, please submit a PR.

Thanks :)

[krakow10]

Great insight, this service file is probably very old.
ovenmediaengine.service.log
Here is the file contents embedded:

[Unit]
Description=OvenMediaEngine
After=network-online.target

[Service]
Type=forking
PIDFile=/var/run/ovenmediaengine.pid
WorkingDirectory=/usr/share/ovenmediaengine
ExecStart=/usr/share/ovenmediaengine/ome_launcher.sh -d
Restart=always
RestartSec=2
RestartPreventExitStatus=1
StandardOutput=null
StandardError=syslog+console
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

I deleted the file and reran sudo make install to get the updated service file, and added a case for debian in the ome_launcher script, but it still says it cannot find any nvidia GPU for the transcoder in the log file.

[krakow10]

I am having the first success on Ubuntu 22.04 after 3 failures (Debian, Arch Linux, Ubuntu 24.04). Ubuntu 22.04 is showing the transcoder startup recognizing the GPU. Using a Ubuntu container is inconvenient for me because the packaged nvidia driver version is not guaranteed to match between the Debian host and the Ubuntu container.

Troubleshooting Ubuntu 22.04:

• installed nvidia-565-headless-server which, at this moment, matches host driver version (565.57.01)
• installed nvidia-cuda-toolkit
• installed ovenmediaengine from source
• nvcuvid fails to load
• installed libnvidia-decode-565-server

Cannot load libnvidia-encode.so.1

Dec 27 05:35:30 ome-ubunter ome_launcher.sh[131213]: [2024-12-27 05:35:30.369] E [ENC-h264nv-t9:131478] FFmpeg | third_parties.cpp:111  | [AVCodecContext: 0x798135400000] Cannot load libnvidia-encode.so.1
Dec 27 05:35:30 ome-ubunter ome_launcher.sh[131213]: [2024-12-27 05:35:30.369] E [ENC-h264nv-t9:131478] FFmpeg | third_parties.cpp:111  | [AVCodecContext: 0x798135400000] The minimum required Nvidia driver for nvenc is 470.57.02 or newer
Dec 27 05:35:30 ome-ubunter ome_launcher.sh[131213]: [2024-12-27 05:35:30.370] E [ENC-h264nv-t9:131478] Transcoder | encoder_avc_nv.cpp:166  | Could not open codec: h264_nvenc (27)
Dec 27 05:35:30 ome-ubunter ome_launcher.sh[131213]: [2024-12-27 05:35:30.370] E [DEC-h264-t256:131448] Transcoder | transcoder_stream.cpp:909  | [#default#hw60/quat(100)] Could not create encoder. Encoder(8), OutputTrack(9)
Dec 27 05:35:30 ome-ubunter ome_launcher.sh[131213]: [2024-12-27 05:35:30.371] E [ENC-h264nv-t17:131479] FFmpeg | third_parties.cpp:111  | [AVCodecContext: 0x798135400400] Cannot load libnvidia-encode.so.1
Dec 27 05:35:30 ome-ubunter ome_launcher.sh[131213]: [2024-12-27 05:35:30.371] E [ENC-h264nv-t17:131479] FFmpeg | third_parties.cpp:111  | [AVCodecContext: 0x798135400400] The minimum required Nvidia driver for nvenc is 470.57.02 or newer
Dec 27 05:35:30 ome-ubunter ome_launcher.sh[131213]: [2024-12-27 05:35:30.371] E [ENC-h264nv-t17:131479] Transcoder | encoder_avc_nv.cpp:166  | Could not open codec: h264_nvenc (27)
Dec 27 05:35:30 ome-ubunter ome_launcher.sh[131213]: [2024-12-27 05:35:30.371] E [DEC-h264-t256:131448] Transcoder | transcoder_stream.cpp:909  | [#default#hw60/quat(100)] Could not create encoder. Encoder(9), OutputTrack(17)
Dec 27 05:35:30 ome-ubunter ome_launcher.sh[131213]: [2024-12-27 05:35:30.371] W [DEC-h264-t256:131448] Transcoder | transcoder_stream.cpp:1137 | [#default#hw60/quat(100)] No encoders have been created. InputTrack(256)

• installed libnvidia-encode-565-server

After this troubleshooting, it seems to be running. Hopefully this can be an intermediate solution while the problems with Debian are sorted through.

[Keukhan]

Debian is not an officially supported OS for OvenMediaEngine, so we don't have a development or testing environment for it. It would be greatly appreciated if you could submit a PR. :)