Take xmalloc into account

[wks]

Ruby wrapped malloc into xmalloc, recording the number of bytes allocated, and use it to guide GC. When using MMTk, we currently do not report the number of bytes allocated by malloc to mmtk-core. As a result, mmtk-core will not trigger GC even when there are significant amount of objects that have part allocated with malloc (such as strings and arrays with the payload allocated with xmalloc). Instead, malloc will fail when running out of memory, and the program will crash. Sometimes (especially when running make install -j while MMTk is enabled) it will exhaust the memory of the entire system. When that happens, the system will become unresponsive, and the OOM killer will start randomly killing processes.

mmtk-core recently gained the ability to count the number of bytes allocated by malloc into the allocated size, via the malloc_counted_size feature. We can make use of it so that we can trigger GC more promptly.

[wks]

The easiest way to test is

./miniruby --mmtk -e '(1..100).each do |i| 100.times do "a"*10_000_000 end; puts "#{i}GB" end'

It will be killed by SIGKILL. However, since all long strings allocated in the loop are garbage, each GC should free up sufficient memory so the program could continue.

[wks]

#8
mmtk/ruby#21

These two PRs added support for obj_free using MMTk's Java-style resurrecting finalization mechanism. We now manually trigger GC when allocated too much memory via xmalloc, using the same heuristics as vanilla Ruby.