Configurable DB Mmap Size for Beacon Node and Validator Client

[rauljordan]

What type of PR is this?

Feature

What does this PR do? Why is it needed?

Currently, we notice that operating a large number of validators has really terrible performance during the first 10-30 minutes of operation, with many deadlines exceeded and long read/write times for db operations. After some investigation into bolt db defaults, looks like they have a very conservative default for the initial mmap size. During startup, bolt dynamically remaps the file to match the resource demand of the software, and during this time, read operations and write operations can have high contention rates between each other. We believe this is the root cause of the degraded performance we see on Pyrmont, and such an issue has also been reported to bolt authors and brought up several times in the past.

This PR creates a shared flag named --bolt-initial-mmap-size which is set to a default of 512Mb

More information

BoltDB options expressly mention the following:

// InitialMmapSize is the initial mmap size of the database
// in bytes. Read transactions won't block write transaction
// if the InitialMmapSize is large enough to hold database mmap
// size. (See DB.Begin for more information)
//
// If <=0, the initial map size is 0.
// If initialMmapSize is smaller than the previous database size,
// it takes no effect.
InitialMmapSize int

Supplemental reading: How databases use mmap