LLVM memory error crash on 2.0

[n8d]

A few times now on Jungle running nodeos 2.0.0-rc2 (B1 packaged version) we have had nodeos crash with just this in the log:

LLVM ERROR: out of memory

Just before the crash our monitoring reports normal memory usage for the nodeos process (4-5GB with ~25GB free) so it doesn't seem to be a memory exhaustion issue. On the last crash we managed to get a core dump and here is a full backtrace:

gdb.log

[swatanabe-b1]

There are two issues here.

• It looks as if too many message_buffers are being created by the net_plugin.
• LLVM is installing its own new_handler which prevents boost::pool from recovering from excessively large allocations.
  @heifner

[heifner]

Here is a backtrace from a test failure
backtrace-eosio-build-19078-amazonlinux-2-nodeos-forked-chain-lr-test.log

[heifner]

The test failure might be unrelated. Steven found a race condition which there is a PR (#8305) up to fix. However, I think the original issue reported here is probably something different.

[heifner]

And Steven found another race condition, fixed by EOSIO/fc#125, which should fix the original reported problem on this issue.

[heifner]

Resolved by #8311