Memory usage

[thedevop]

I did a memory usage comparison between 2.1.6 and 2.2.3 (post dedicated write chan per client). The setup is ~140K concurrent clients, each client send ~2KB to the broker every minute.

	2.1.6	2.2.3 (128)	2.2.3 (1024)
RSS GiB	8.5	10.5	13.5

2.1.6: FanPoolSize = 512, with FanPoolQueueSize = 1024
2.2.3: MaximumClientWritesPending in parenthesis above

The difference is larger than expected, so I did a heap profile on it, and found currently the majority of the memory allocation is due to NewReader and NewWriter (both defaults to 4096B)
https://github.com/mochi-co/mqtt/blob/44ce819318f67eee790d5126355f1229e57a8d1a/clients.go#L172
We make this configurable, I can submit a PR if you like.

[mochi-co]

Good find! This should now be configurable in v2.2.6: https://github.com/mochi-co/mqtt/releases/tag/v2.2.6 - let me know if it works as expected!

[thedevop]

Preliminary usage in staging looks promising. I'll report back the numbers when it's in production.

[thedevop]

Thanks @mochi-co. With 2.2.6, with of ClientNetWriteBufferSize=ClientNetReadBufferSize=2KB, MaximumClientWritesPending=128, the memory usage is down to similar size as 2.1.6.