VM crash when reaching maxOldSpaceSize limit with small allocations and no scavengeable objects

[daniels220]

I am trying to use the maxOldSpaceSize VM parameter to provide a low-overhead notification when my application exceeds a certain amount of memory usage, but when I try it out, it seems that the VM is unable to actually recover gracefully from a low-space condition due to hitting the limit unless the allocation that pushes it over the limit is large enough that smaller ones can subsequently succeed. In other words:

"Set a value that will predictably trip with a reasonable amount of allocation:"
Smalltalk vm maxOldSpaceSize: Smalltalk vm oldSpace + 10e6.
"A single large allocation fails and an error is raised from #growMemoryByAtLeast:"
[ Array new: 10e6 asInteger. true ] on: OutOfMemory do: [ :ex | false ]. " => false"
"But allocating in a bunch of small chunks just crashes the VM:"
[ (1 to: 10000) collect: [ :each | Array new: 1000 ]. true ] on: OutOfMemory do: [ :ex | false ].

Fortunately, it seems like this worst-case scenario only occurs when all the allocations are still referenced—if I add some garbage allocations:

[ (1 to: 10000) collect: [ :each | 5 timesRepeat: [Array new: 1000]. Array new: 1000 ]. true ] on: OutOfMemory do: [ :ex | false ].

Now I get the false result as expected—though I tend to also get an ED emergency debugger immediately after, but I can proceed from it and the system will collect the array and return to seemingly-normal operation. However if I limit the amount of garbage by removing the timesRepeat:, so there is only one garbage array per non-garbage array, and I run it twice in a row (remember, all memory is released at the end of each run), I generally get a crash. If I am generating a lot of garbage, I can still get a crash by doing this several times (as an application might if it were running very close to the memory limit, generating lots of garbage but also slowly creeping up in overall usage). Sometimes the crash does not occur until the system is idle for some time after running the snippet.

For my use-case I hope that since I will take action when the limit is reached the first time, I should be able to avoid crashes so long as there is sufficient garbage available to reclaim when the limit is hit, but I worry that I will get unlucky just after a scavenge every once in a while...