Chunking in frequency may cause excessive memory use

[JSKenyon]

• dask-ms version: master
• Python version: 3.8.8
• Operating System: Pop!_OS 18.04

Description

While running my small example case, chunking in frequency causes bizarre memory behaviour.

What I Did

I don't have a reproducer, but it boils down to reading a measurement set while chunking in frequency.

What happened?

The memory usage grows, seemingly without bound, over time. Contrast this with reading from zarr:

Why?

My best guess is that this is a problem with the way casacore caches data for tiled columns. Dask does not process chunks in a consistent order - I believe that this leads to the all the tiled columns being cached in their entirety. This may only happen when the frequency chunk is smaller than the frequency tiling of the MS.

[sjperkins]

Related to:

• xds_from_ms memory bloat with frequency chunks #92
• Work around casacore getcolslice caching #107

Also, recalling our conversation, changing the number of frequency chunks did not increase or decrease the amount of memory used in the CASA Table case.

[JSKenyon]

Closing as this is mainly an edge case. This should only happen when data is poorly tiled/very small.