[BUG] Memory Leak for read_raw_edf with TAL data (~20 GB)

[alexrockhill]

Unfortunately, I can't share a minimally reproducible example because the problem stems from having a very large file. Interestingly, this was not a problem on a MacOS system, the file loaded with preload=False just fine. The issue is that when I go to read the file, I get > 16 GB of RAM usage. Here is the output of mne sys_info on the system:

Platform:         Linux-5.4.0-110-generic-x86_64-with-glibc2.27
Python:           3.10.4 | packaged by conda-forge | (main, Mar 24 2022, 17:38:57) [GCC 10.3.0]
Executable:       /home/alex/software/anaconda3/envs/mne/bin/python3.1
CPU:              x86_64: 4 cores
Memory:           15.5 GB
mne:              1.1.dev0
numpy:            1.21.5 {blas=NO_ATLAS_INFO, lapack=lapack}
scipy:            1.8.0
matplotlib:       3.5.1 {backend=QtAgg}
sklearn:          1.0.2
numba:            0.55.1
nibabel:          3.2.2
nilearn:          0.9.0
dipy:             1.5.0
cupy:             Not found
pandas:           1.4.1
pyvista:          0.33.3 {OpenGL 4.5.0 NVIDIA 515.43.04 via NVIDIA GeForce RTX 2070/PCIe/SSE2}
pyvistaqt:        0.7.0
ipyvtklink:       0.2.2
vtk:              9.1.0
qtpy:             2.0.1 {PyQt5=5.12.9}
ipympl:           Not found
pyqtgraph:        0.12.4
pooch:            v1.6.0
mne_bids:         0.11.dev0
mne_nirs:         Not found
mne_features:     Not found
mne_qt_browser:   0.2.5
mne_connectivity: Not found
mne_icalabel:     Not found

One of the things I'm curious about is what's going on here: https://github.com/mne-tools/mne-python/blob/main/mne/io/edf/edf.py#L310-L337, it looks like the only thing being returned is the tal_data but yet there are these other computations with the channel data. These don't seem to get returned and I'm not sure if they are supposed to modify something in place or what. Maybe @sappelhoff or @cbrnr knows?

[cbrnr]

Could it be that I just solved this with #10638?

[alexrockhill]

Uh oh, I still got the memory spike with the latest code...

[cbrnr]

Does this mean you see absolutely no difference in the memory profile? Or did it at least drop significantly? If it's just because of a very large EDF file, you can adapt the example code in #10634 (#10634 (comment)) to generate a larger file.

[alexrockhill]

It's still too large of a spike for my 16 GB of RAM but I'm having a bit of a difficult time memory profiling because I have to kill the process before the spike overloads the RAM and crashes the computer...

[cbrnr]

It's a bit hard to say without the file or a reprex. Can you try to generate a large EDF file to see if this has the same problem? Then at least other people can reproduce the issue.

[alexrockhill]

Nevermind, I think the Ubuntu trash collector just works more on-demand and when I saw the memory trending up toward maxing out the RAM, I killed the process but I let it go a bit longer the next time and it got cleared. Thanks for solving this @cbrnr!