Better memory management for zap logger

[nagl-temporal]

Is your feature request related to a problem? Please describe.
The zap framework can consume an unbounded amount of memory, and often consumes more memory than it "needs to" due to naive use of sync.Pool.

For context, see this golang issue for a discussion of best (and worst!) practices around sync.Pool. Zap follows the bad practice of putting variable-sized buffers in its pool.

I've seen this cause bad behavior in temporal when it is frequently logging small lines but infrequently logging large lines. This pattern leads to an unnecessarily large zap buffer pool (in practice, I've seen up to 2.5gb). The golang issue discusses this scenario specifically here.

Describe the solution you'd like
See below - it's nothing but alternatives!

Describe alternatives you've considered
There's a lot we could do about this. Off the top of my head:

• Dedup recent logs with long lines (e.g. ones with stack traces)
• Commit any/several of the suggestions from the golang issue to zap:
  • Sometimes don't return large buffers to the pool.
  • Use a tiered pool (small vs. large).
  • Implement a real upper bound on pool size (and don't return buffers over that bound to the pool).