Why does the nv_open_q process in my system take up too much CPU without running additional processes outside the system?

[Turbolento]

OS: unraid 6.12.8

Graphics card: RTX 3060 Laptop GPU

Open Source Driver: v550.40.07

I will provide more information if necessary.

[Turbolento]

Before installing the Open Source Driver, the CPU usage was below 5%. After installing the Driver, the usage has been around 25%.

[aritger]

That is surprising. open_q was added fairly recently to better parallelize user-space processes opening /dev/nvidia* files, but it should be idle the vast majority of the time.

What else is the system doing when you see this? E.g., do you have an X server running? Or wayland compositor? Or nvidia-persistenced?

An nvidia-bug-report.log might help, too.

[Turbolento]

@aritger Thank you for your answer. After 24 hours, the CPU usage should have returned to the normal range. However, I will continue to observe in the future. If the CPU abnormality occurs again, I will record and upload nvidia-bug-report.log

[xeekworx]

I'm on UNRAID 6.12.10 and I am noticing this going on right now. It's pegging 3 of my logical processors.

[xeekworx]

There shouldn't be anything using this GPU right now; I do not see any apps utilizing it in the GPU stats right now, no one is watching Plex videos where I have it configured to be used for encoding right now.

[xeekworx]

nvidia-bug-report.log.gz
The nvidia-bug-report.log for my system. I actually couldn't kill that process for some reason so I'm about to reboot the system.

[Masses3581]

I could copy&paste everything from @xeekworx. I have the same version of Unraid Server 6.12.10, same NVIDIA driver version 550.67, same problems.... :-( The nv_open_q thread is boosting up my CPU system, normally it should stay quiet.
The only difference i see ist: i am using emby server instead of plex.
nvidia-bug-report.log.gz

I hope this helps maybe to identify the problem source. Thank you for help.

[mtijanic]

As Andy says, this is really surprising, probably a case of some userspace process misbehaving. We'll dig through the attached bug reports, thanks!

In the meantime, could someone please try using the NVreg_EnableNonblockingOpen=0 module param and report back whether it makes the issue go away?

[JsGraaf]

Fixed it for me. For reference, I'm on Fedora 40 with 6.8.10-300.fc40.x86_64 and Nvidia kernel 550.78 with a GTX1650 Max-Q

[amrit1711]

Hi All,
Request you to please trace open calls with bpftrace as per below example.
You can download the latest binary from https://github.com/bpftrace/bpftrace/releases.

bpftrace -e 'kprobe:nvidia_open { printf("[%llu] nvidia_open by %s\n", nsecs, comm); }'

[kbareis]

For folks on Unraid, I think I figured it out. My nv_open_q only pegs 50% CPU when I am on the dashboard. This is likely due to having the GPU widget enabled so I can see process, memory usage, and such. What is funny is as soon as I move to another tab, the process drops back down to almost 0.

[luyufan498]

Same to me, seems that the dashboard is the key ...

[luyufan498]

The nv_open_q uses 50% of 8700K..

switch to the other panel does fix this....

[CorneliusCornbread]

Seeing this happen with Unraid as well, nv_open_q using tons of CPU usage while Plex is idle. Insanely obnoxious, driver version 560.28.03, Unraid 6.12.11

[ich777]

The maintainer from the Nvidia Driver plugin for Unraid here... :)

Can you guys please check the interval in your GPU Statistics plugin and see to what it's set?
If you only see this behaviour on the Unraid Dashboard then it's most certainly caused by the GPU Statistics plugin since if you set the interval in GPU Statistics to 2 seconds it will call nvidia-smi every 2 seconds which causes spikes (this is a known thing), I would recommend to set the interval to 5 seconds or even higher.

Please also note that any open Unraid Dashboard page, regardless on which Device, will cause these spikes, so don't forget to check your other Devices if maybe a Unraid Dashboard page is open somewhere.

[ich777]

Thanks @BlueGoliath

[SimonFair]

Which links to this Github:

https://github.com/b3rs3rk/gpustat-unraid

I took over the plugin, but the original support post still has the original developer. Here is the current repo https://github.com/SimonFair/gpustat-unraid

The process still run nvsmi. Can you provide guidance on correct process to get the stats and I can look to see if the plugin can support that way.

[mtijanic]

What @BlueGoliath says about NVML is true, however we do recognize that running nvidia-smi and parsing the result is often much easier. We are looking into ways of making that flow easier to use and less prone to unexpected perf issues.

@SimonFair, thanks! Looking at the linked code the usage of nvidia-smi seems reasonable, but I think whatever is calling into that PHP might be doing so way more than you'd expect. There seem to be two different .page files that use different calls to setInterval() to set up the refresh rate, and I got lost trying to follow it all in an unfamiliar codebase. I'd suggest you just double-check how often you are actually calling nvidia-smi (both for getStatistics() and for getInventory() families of functions). There might be something in the dashboard infrastructure itself that is calling you more than you'd expect.

My suggestion is to make a wrapper script nvidia-smi-cnt.sh that looks something like:

#!/bin/bash
if [[ ! -f "/tmp/nvsmi-calls-cnt" ]] || ! [[ $(cat "/tmp/nvsmi-calls-cnt") =~ ^[0-9]+$ ]]; then
    echo "0" > "/tmp/nvsmi-calls-cnt"
fi
CNT=$(echo "$(cat "/tmp/nvsmi-calls-cnt") + 1" | bc)
echo "$CNT" > "/tmp/nvsmi-calls-cnt"
echo "Total nvidia-smi calls: $CNT"

nvidia-smi $@

and then switch to calling that instead, then see how fast the counter gets incremented.

By the way, @ich777 , looking at your repo uses of nvidia-smi, three out of the four can just be replaced with cat /sys/module/nvidia/version. For the last one, using nvidia-smi is reasonable if you need the UUID, I don't think it's easily accessible via sysfs.

[SimonFair]

getInventory()

This only gets call from the settings page file. the dash board will call the command every x seconds depending on the refresh the user specifies. My test system has multiple gpus

[SimonFair]

You can find the header here. While I hate Python, it would probably be easiest since bindings already exist unless you're willing to put work into a C server or some other custom solution.

thanks will review I dont think python is the best approach but will look into the api. Thanks for the pointers.

[PhyoeBlitz]

I'll be too late when i write this.By the way,
did you open nvidia-smi like i did?
when i run nvidia-smi with
watch -n 1 nvidia-smi
it consumes 50% of cpu. When I close it, the CPU usage drops to 0%.