Add a fine grain memory usage tracking / break down of memory usage *within* each operator

[rluvaton]

Is your feature request related to a problem or challenge?

Yes, debugging memory problems are hard, when running DF in production and the memory pool is unable to grow the memory it will return ResourcesExhausted error or panic depending on the memory pool implementation, however we don't know what takes all the memory.

In my case I had to manually patch DataFusion row_hash file to print on error what every accumulator takes and the internals

Describe the solution you'd like

I want to add new API - that has size function (not required just because I think it should be combined with explain_memory) and explain_memory function or something (similar to Debug trait) to get string with breakdown of the size with every thing that takes memory.

So for GroupedHashAggregateStream it would say how much group values takes, and for each accumulators it would call the explain memory on each

Describe alternatives you've considered

No response

Additional context

No response

[zheniasigayev]

Within the DataFusion CLI there is a flag called:

--top-memory-consumers <TOP_MEMORY_CONSUMERS>
    The number of top memory consumers to display when query fails due to memory exhaustion. To disable memory consumer tracking, set this value to 0 [default: 3]

This helped me identify large consumers of memory in: #16776. Perhaps the same solution can be implemented for your request.

[rluvaton]

The problem with that is there is no breakdown on what the memory is actually spent on in each consumer

so if I have a aggregation that takes a lot of memory I won't know about it

[alamb]

I agree more details are good -- I think the trick will be figuring out how to do so in such as way that is reasonable to maintain as well as doesn't slow down the normal operation

[alamb]

Also possibly related:

• [DISCUSSION] Memory accounting model discussion #16841

[alamb]

• Related Add Memory Profiling Functionality #14510

[rluvaton]

With this feature, issues like this are much easier to debug:

• Disproportionate memory use for DISTINCT ON query #17169

[alamb]

@kosiew has a PR that adds this capability -- please check it out if you have time

• Add memory profiling support to DataFusion CLI and memory pool metrics #17021

Basically it uses the existing TrackConsumersPool that was contributed by @wiedld to get memory usage per operator

Maybe we need to add some more documentation / examples of how to use that API

[wiedld]

@alamb -- as for more documentation, do you means adding code examples with the TrackConsumersPool docs? And/or in the datafusion-examples?

[alamb]

Specifically I think @rluvaton is requesting something that is already possible with TrackConsumersPool -- and if that is the case, then we could potentially resolve the ticket by adding some more docs / examples to the TrackConsumersPool.

We might add an example (ideally in the docstrings) of how to get the largest consumers of a plan / sample the current memory users while the plan is running