ISSUE-9562: maxMessageBufferSizeInMB is not working?

[sijie]

Original Issue: apache#9562

---

After some load testing on my pulsar cluster, thr maxMessageBufferSizeInMB configuration does not protect the broker from being OOMED, im publishing to a single topic about 150 mb/s (5mb message size) and my direct memory is going up until the broker crashed because of bookie disk latencies, I would expect some throttling on the producers side using this configuration released in 2.5.1 apache#6178

I allocate 8 gb of direct memory to the broker and I configured it to a range of 1-4096

Steps to reproduce the behavior:

1. Use the default maxMessageBufferSizeInMB (1/2 of direct memory)
2. Load the cluster with 5mb messages untill bookies disk have high latencies
3. wait for broker to crash

I would expect producers to get throttled untill the broker does succeefully get acks from the qa of bookies.

The holes in the graphs describes when the broker failed and also when I tried to tune this configuration

• OS: Centos 7.8

You can see more details about it in this slack thread https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1612886327337200?thread_ts=1612886327.337200&cid=C5Z4T36F7

Btw, it can also happen with small messages