Ensure blocks are full when mempool exceeds block size

[cmwaters]

on July 30th, we observed increased transaction demand. Among other things, we noticed that while the mempool was hovering around 32MB in size, we were still producing blocks sufficiently less than the 2MB max

Above you see occasional spikes where a 2MB block is committed, but this is not consistently full as would be expected.

We should investigate why this occurs and find a way to ensure that when the mempool is significantly large we are consistently producing full blocks

[evan-forbes]

to complicate this further, we're not seeing the above plot on all nodes. some nodes are normal

[cmwaters]

The most recent plots show that the mempool size is now similar to the block size but it's weirdly not following it. It's as if there's a large transaction that is added and is eventually evicted but we don't see it committed in a block