Consider replacing (or adding) jemalloc with v3.6.0

[ashie]

From fluent/fluentd#1657

Some users reported that jemalloc v4 or v5 consumes excessive amounts of memory and v3.6 quite reduces it. It seems a known issue as follows:

• https://bugs.ruby-lang.org/issues/14718
• https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html
• https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html

Do not use jemalloc 4.x with Ruby. It has a bad interaction with Transparent Huge Pages that reduces the memory savings you’ll see. Instead, use jemalloc 3.6. 5.0’s performance with Ruby is currently unknown.

Replacing (or adding) jemalloc with v3.6.0 isn't hard, but I think there is a points to be considered.
Is v3.6 still supported and can we get security fixes (it seems stable enough though)?
It's also concerned in the above ruby-lang's issue:

https://bugs.ruby-lang.org/issues/14718#note-37

it will be harder to get support from the jemalloc team if we are not on 5.0.1. 

Enterprise users are sensitive in security issues so it would be better that we have evidence about it. According to jemlloc's web site, it still seems be supported:

http://jemalloc.net/

stable-3/stable-4: The stable-* branches are bugfix-only branches based on the last releases for previous major versions. The versioned release history is linear, but relevant fixes may continue to be applied to the stable-* branches indefinitely.

[repeatedly]

NOTE: We can control jemalloc behavior via MALLOC_CONF. So if we can reduces memory usage by using jemalloc options, using latest jemalloc is better. jemalloc 3.6.0 is released in 2014 and it doesn't consider recent systems and hardwares. Of course, using 3.6.0 as temporary solution is good for users.

[ashie]

Thanks for the information! I'll check it.

[dmitrytokarev]

@ashie Do Docker images need to downgrade jemalloc as well as upgrading ruby?
https://github.com/fluent/fluentd-docker-image/blob/master/v1.13/debian/Dockerfile#L35-L36
https://github.com/fluent/fluentd-docker-image/blob/master/v1.13/debian/Dockerfile#L4

[ashie]

I don't consider downgrading it for Docker images for now.
Not always needed to downgrade if most users doesn't hit the memory usage issue.
Generally speaking, as we discussed above, using an old release isn't recommended.

As described in the references in #305 (comment), downgrading jemalloc might not be only way to reduce memory usage. So we'll continue investigating to find a better way. For example, Heroku uses MALLOC_ARENA_MAX=2 to achieve the purpose: https://devcenter.heroku.com/changelog-items/1683

For td-agent, we downgraded jemalloc to verificate one of such ways.
If this way is verified as "most better way" (never the best because it's an old release), we might propagate it to Docker images too.