large memory usage / python not freeing soon enough?

[teuben]

Describe the bug
I've mostly used spyder. But several times I run into a loaded memory (32GB physical, 54GB swap) on moderately large SDFITS files, in my case today a 23GB sdfits file. The machine started swapping in the middle of a (dysh) zoom session.

For now this is a placeholder for what might be a python of sdfits loader problem, but love to hear more experiences if others are running into this problem. I'm certainly fairly frustrated with this.

How to Reproduce
I will work on getting a reproducible script, but obviously will depend on lot on the users' memory size (and possibly swap)

Environment

• Dysh version
• Python version 3.12 (i believe)
• OS Ubuntu 22

[teuben]

It can help to (at least in spyder) to see which variables you have, and how much memory python is taking up. Sometimes a simple

     del sdf5

to remove an object can reclaim memory. It should of course, but doesn't always seem to do it right away.