Memory issues during genesis sync

[michaelsproul]

Description

MigaLabs and now Gnosis have both reported Lighthouse using a lot of memory during genesis sync. We thought it might have been #2944, but with that issue resolved the problem is still occurring.

Looking at some metrics on the affected Gnosis servers it seems not to be a memory fragmentation issue, as the non-mmap memory is high (~10GB) and the free block memory is low (<800MB). I think to get to the bottom of this we would need to run lighthouse under heaptrack while syncing from genesis on mainnet or Gnosis chain.

Version

v3.2.1

Workarounds

Use checkpoint sync. It's a lot faster in multiple ways and unaffected by this issue.

There will be some checkpoint sync servers set up for Gnosis chain before the Gnosis merge.

[michaelsproul]

May be alleviated by #3697. It would be interesting to compare GNU malloc vs jemalloc during genesis sync