Does arrow-rs support tracking buffer memory usage?

[jp0317]

Hi, i'm wondering if there's an existing interface to track buffer memory usage within arrow-rs. In C++ there's an arrow::MemoryPool, e.g., in page decompression it's used for decompressor buffer while in rust the buffer is from Vec. Tracking memory usage can be helpful for cases where memory control is desired.

[tustvold]

We provide methods to get the allocated size of buffers in arrays and these can be hooked into a memory tracking system. DataFusion has examples of this. We don't provide anything first-party in part because Rust doesn't have stable support for custom allocators which would be the ideal way to provide this functionality.

[jp0317]

thanks for the rely!

get the allocated size

would this be too late? i.e., it seems better to prevent insane huge allocation by checking the size to be allocated before real allocation.

[tustvold]

We support tracking memory usage, fallible allocation could be achieved by overriding the global allocator.

See also #3960 and #6439