OutOfMemory exception with low system memory usage

[mskarlin]

We're struggling to diagnose an issue where we are instantiating several Index objects (built on the same shared directory) in asynchronous coroutines, and setting the following stack trace:

ValueError: Failed to open file for read: 'IoError { io_error: Os { code: 12, kind: OutOfMemory, message: "Cannot allocate memory" }, filepath: "<index file path>" }

We have a relatively small index (~5Gb), and even with several instantiated Index objects, the system RAM usage is low (on a machine with 60Gb of memory, we're only seeing ~3Gb used). But when running with 4 concurrent coroutines we see the above issue.

In trying to reproduce, I made a tiny index with ~100 files, and ran 100 simultaneous coroutines, each of which with their own Index object, and was not able to reproduce the error.

Do you have any suggestions on what to investigate next?

[cjrh]

We're running with thousands of different indexes on a shared box, but we're using process pools instead of your async handling. We don't see memory growth in these long lived processes (that access different indexes over time), but to be fair we have not yet upgraded to 0.22.

As for suggestions, this might be a good application of the FIL memory profiler: https://pythonspeed.com/fil/docs/index.html

In essence, you can run your application in a staging environment with a copy of your "real" index, do what you need to do to make memory grow, and then when memory is high, kill the process. The memory profiler will dump a file showing where allocations were held. You get a flamegraph view:

https://pythonspeed.com/fil/docs/interpreting-output.html

[cjrh]

It would also be good document more specifics about your situation, i.e., package version, platform, etc.

[cjrh]

There was an actual memory leak a few versions back, in PyO3: PyO3/pyo3#3285. So it is important to know which version you're using. There are also a few other memory leak issues on the PyO3 tracker, so you might have a look there too to see if anything might be affecting you.