Cleanup memory pool component and add large size. Clarify user customizability

[lifflander]

What Needs to be Done?

Pool/MemoryPoolEqual should be cleaned up. It's unclear what pieces the user can customize and how they can do that. MemoryPoolEqual should have a better name also like FixedSizePool

[pnstickne]

It would be nice if thePool interface ALSO worked when it was disabled (ie. using malloc/free directly), such that there would not need to be difference in code usage sites. There are at least 4 deviation sites in active.cc (as an example) based on the check status.

I suppose it's not "really" a pool then, although not sure such distinction is very important from usage sites.

[lifflander]

That's a great point. We should make it work either way so we don't have to have that ifdef.

Also, maybe we should create an allocator that follows the std::allocator spec for users that is backed by the pool. That is technically the correct way to change allocation I think.

[lifflander]

@bradybray Can you work on this next? I want to make the memory pool is VT usable by users (so they can create their own FixedSizePool (currently MemoryPoolEqual). Then, I want to add a large size to the pool. Also, it would be nice to do some benchmarking with small messages to see if it helps performance.

[lifflander]

• Like Paul mentioned, I also want to make all the regular mallocs that get ifdef'ed out (like the the ActiveMessenger) always go through the pool interface.
• In addition, I would like to add diagnostics for the pool. CounterGauge on allocations so get can get the average size and number of allocations

[lifflander]

When we have some extra time, we can restart this one.