Enhance ztunnel heap instrumentation

[bleggett]

Right now ztunnel exposes gperftools endpoints for both profiles and heaps, as well as a standard pprof profile.

ATM the gperftools endpoints don't work very well, or at all.

We should probably consolidate on 1 profiler and one heap dumper, and make sure they both work - whether that's gperftools or something else (jemalloc was mentioned).

Whatever we DO pick needs to work equally well on arm64 and amd64, and needs to support remote analysis, ideally.

• Fix or replace gperftools to enable heap dumps: Remove gperftools, use jemalloc #865
• Investigate building with/enabling jemalloc by default, at least for non-release builds, possibly for release builds if it is not a perf hit (e.g. at one time Facebook left jemalloc on permanently in prod due to nonexistent perf hit: https://www.magiroux.com/rust-jemalloc-profiling/)

[howardjohn]

https://github.com/polarsignals/rust-jemalloc-pprof or https://github.com/mkmik/heappy/ seem viable.

Prototype with heappy: https://github.com/howardjohn/ztunnel/pull/new/exp/heappy

[bleggett]

Yep, using https://github.com/polarsignals/rust-jemalloc-pprof, seems to work well enough.

[ilrudie]

should #865 close this one out?

[bleggett]

@ilrudie In #865 we discussed using jemalloc as an always-on thing, or at least enabled by default for non-release builds.

I added that as another ticket item here, so we can keep this open for that.

[daixiang0]

seems the second task has be completed, maybe we can close it.