GPU utilization plot reports repeated GPUs

[mrocklin]

@quasiben and I were playing around with his 2-gpu system, and were surprised to see four GPUs in his dashboard plots. I suspect that each worker was separately reporting the metrics for both GPUs on the system.

The relevant code for this is here:

distributed/distributed/diagnostics/nvml.py

Lines 6 to 12 in ecaf140

	def _pynvml_handles():
	global handles
	if handles is None:
	pynvml.nvmlInit()
	count = pynvml.nvmlDeviceGetCount()
	handles = [pynvml.nvmlDeviceGetHandleByIndex(i) for i in range(count)]
	return handles

Perhaps pynvml does not respect CUDA_VISIBLE_DEVICES? Should we filter this on our own?

cc @rjzamora @jacobtomlinson

[trivialfis]

Hi @mrocklin , I'm using LocalCUDACluster from dask-cuda and can't find the utilization information on dashboard. Is there any document on how to enable it?

[quasiben]

@trivialfis you may need to upgrade the dask lab extension if you are using jupyter lab (recommended).

You can also find it directly on the dashboard at :8787/individual-gpu-utilization

[trivialfis]

@quasiben Thanks for the reply. The direct way works perfectly for me.

@mrocklin

Perhaps pynvml does not respect CUDA_VISIBLE_DEVICES?

If memory serves, you are right that nvml doesn't respect CUDA_VISIBLE_DEVICES, so pynvml won't be affected by this env.

[trivialfis]

Related #3808 .

[jacobtomlinson]

I think this has been resolved by #3810