feat: profiling for GPU usage and throughput metrics

[jayavenkatesh19]

A useful feature would be to implement a rapids profile command to provide metrics for python scripts similar to the jupyterlab-nvdashboard for non-Jupyter users.

A few design choices to think about are:

1. Real-time metrics similar to the nvdashboard vs metrics written to a file to avoid interfering with the outputs of the script
2. Can existing tools such as nsight be integrated for this purpose
3. At what level of the code should this profiling be done in order to better identify bottlenecks, maybe as decorators for important functions?

[ncclementi]

For the case of cudf.pandas and cuml.accel, could this be just a hook to the cudf.pandas.profiler or the upcoming cuml profiler.

[jacobtomlinson]

I played around with this kind of thing in Dask a few years ago (blurry gif). Would be fun to try and recreate nvdashboard in the terminal with rich and textual.

[jayavenkatesh19]

I worked on an initial design for this today. I borrowed a lot of inspiration from jupyterlab-nvdashboard, with the aim of capturing the same metrics in a terminal based panel.

The idea for the design I had is that users would run the rapids profile command in the terminal, and this will start a rich dashboard that polls GPU and CPU metrics every 1 second while the user runs the code elsewhere (in a different terminal)

This is how it looks in the initial design:

Let me know if you have any ideas on how we can improve this, and we can add those to this initial version.

The code for the initial design is on the dashboard branch of this repo

[jacobtomlinson]

This is really nice!

Some high level feedback:

• There are a lot of table borders. I would potentially make the table within the GPU Status section borderless. I would also drop the border on the top section.
• The Interconnect section has a lot of dead space. Could it be streamlined?
• Let's avoid the word profiler, it suggests we are measuring/tracing the code, which we are not. The word "Dashboard", "Resources" or "Stats" may be better.

[betatim]

This looks super cool! I agree that "profiler" is probably not the right word here, I like dashboard.

Just for my education: it looks like it shows roughly the same information I could get from watch -d1 nvidia-smi. Is that because "This is a quick prototype Tim, we haven't built everything!" or is it because that is kind of all the info you can show?

One thing that I like from nvidia-smi is that I can see the processes as well. Maybe something to show here too?

And while thinking about processes and profiling: Is it possible to show some of the information that you see in Nsight via https://nvidia.github.io/NVTX/python/index.html? Having a very simple way of seeing some of that information in a console would be useful. I think. At least so far I've never made use of the NVTX annotations and NSight because it seems like I'd have to spend a day or two figuring out how to hook it up to a process running on a remote machine. So a lightweight console version would be very cool.

I am here all the time in case you have another small project that needs its scope massively expanding ;)

[jayavenkatesh19]

@betatim the work so far is definitely more on the side of a quick prototype rather than a comprehensive solution. The main objective of such a feature in a cli tool for RAPIDS would be more towards helping users identify potential bottlenecks and improve RAPIDS code rather than just track bare GPU usage metrics, which as you rightly pointed out, are already available by polling the nvidia-smi.

The real value would be to add some kind of profiling here, either from the cudf.pandas.profiler or the NVTX annotations in NSight, with a fallback to still having GPU metrics displayed in case the end user hasn't set up any profiling in their code.

[betatim]

Sounds like the goal is to have something between NSight and nvidia-smi. Thinking about something you can't do with nvidia-smi that I miss: seeing which (high level) part of my program is being executed while watching things like data transfer rate, GPU usage, memory usage and CPU usage. For example a cuml script will have phases where it reads from disk/network, then does some preprocessing and then some estimator fitting. Depending on which part is executing, you should see different things for the observables.

Maybe a simple (haha...) thing to do is to interrupt the python process at regular intervals, look at the stack and use that to provide the user of the dashboard with information about "where in the script" execution is. With "interrupt python and look at the stack" I'm thinking of how py-spy (and friends) work. Hence the "haha..." after "simple". But at least there is prior art ;)