Log when spilling from GPU memory to host memory and from host memory to disk happens.

[EvenOldridge]

Add logs that can be turned on/off with dask.config.set that log when spilling occurs.

Spilling has such an adverse effect on perf that it's important to highlight to our customers when it happens. Probably also useful to add guidelines for improvement the first time it occurs.

Something along the lines of:

"Worker X | Task Y hit GPU memory limits. ___ MB? spilled to host. This can impact overall task performance."
"Increase your number of GPUs or GPU memory to avoid this issue. Webpage provides guidance for configuring your cluster and workflow to avoid spilling" (only on first error of this type?)

and

"Worker X | Task Y hit host memory limits. ____ MB? spilled to disk. This will significantly impact performance."
"Increase on host memory to avoid this issue. Webpage_ provides guidance for configuring your cluster and workflow to avoid spilling" (only on first error of this type?)

Obviously these need to be refined and workshopped, but capture the gist of what I'd like to see.

[quasiben]

We can do a little bit of this in dask-cuda but the host memory to disk occurs in dask/distibuted and in zict which I don't think we log. @pentschev is that correct ?

[pentschev]

No, we entirely replace Dask's spilling mechanism with our own, so we can log things without messing with Dask core. We might need to add a new configuration flag in Distributed though, I'm not sure if we can extend to have our own configs in dask-cuda without adding that to Distributed, can we @quasiben ?

[quasiben]

We could have our own config but I would caution against this. It makes sense for things like dask-jobqueue, dask-cloudprovider, etc, but we already have so much in dask-core (ucx and rmm config) that I would think we would want to keep those things, which mostly dask-cuda uses, centralized. What kind of config flag were you thinking about it

[pentschev]

We could have our own config but I would caution against this. It makes sense for things like dask-jobqueue, dask-cloudprovider, etc, but we already have so much in dask-core (ucx and rmm config) that I would think we would want to keep those things, which mostly dask-cuda uses, centralized. What kind of config flag were you thinking about it

I was thinking of something like logging.dask-cuda.spilling because that's gonna be used for dask-cuda only. I don't have an issue with that staying in distributed, but it feels really weird to have a flag there that will only be used by dask-cuda and no effect whatsoever in distributed alone. The other flags that you mentioned are both used primarily by dask-cuda users, but they were both added there because we needed them to configure dask-scheduler, before that was needed we had it all in dask-cuda.

[jakirkham]

Are we sure this couldn't also be used by distributed? It sounds like it could be, in which case maybe it would be better to use a generic name

[pentschev]

Are we sure this couldn't also be used by distributed? It sounds like it could be, in which case maybe it would be better to use a generic name

I'm not saying it couldn't be, but rather I think it wouldn't, at least immediately. I don't oppose to having that functionality in distributed as well, but someone's gotta do the work there, which I assume won't happen unless someone really needs that.

[pentschev]

@EvenOldridge I had a discussion offline with @quasiben around this request. I believe what your users want is some kind of log that displays directly on the client console or some other user-accessible place without much change, is that assumption correct?

Today, there's no functionality like that in Dask, either Dask-CUDA or Distributed, and it's not clear to us whether this should ever be their responsibility to do so. One idea we had is that users could register an async PeriodicCallback function that would call client.get_worker_logs(), parse the logs to get only the relevant part and handle output/printing where most convenient for the application at hand. Would that work for you?

Second, we still need to log the spill calls, and this is Dask-CUDA's responsibility -- potentially Distributed in the future. This is something that we'll start by logging only in Dask-CUDA for now and generalizing later.

[pentschev]

I opened #442 with an initial attempt to solve this, covering only LocalCUDACluster at the moment, but we can easily extend that if that's helpful to solve the problem @EvenOldridge has been asked.

[github-actions]

This issue has been marked stale due to no recent activity in the past 30d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be marked rotten if there is no activity in the next 60d.

[pentschev]

This has been addressed in #442 , closing.