Dynamic memory reduction of ~13% (up to 19% observed) by tweaking MALLOC env vars

[BobbyRBruce]

This information comes from Prof. W.B. Langdon from University College London. He has been working on techniques to tune/optimize various parameters to improve the running of software and has been using gem5 as a target application. He has asked me to make the community aware of his findings and wonders if there could in some way be incorporated into the project.

His work has found that the Malloc Tunable Parameters have suboptimal defaults for using gem5, resulting in an overuse of Dynamic memory. The default parameters result in overuse of heap memory at the expense of MMAP which is seldom used. By optimizing the Malloc parameters he found we can increase MMAP usage / reduces heap usage which, all in all, results a dynamic memory size reduction.

Running a few programs (of varying sizes) in gem5 he found dynamic memory savings of between 10% to 19%, with around 13% being typical (as reported by memalloc_info()). In one big execution of an RNAfolding applicaiton he found MMAP usage jumped from just twice per run to 353 times due to his otpimizations. He also observed a tiny speedup on gem5 execution of about 0.5%.

The optimal environmental parameters he found were as follows:

setenv MALLOC_TOP_PAD_ 646
setenv MALLOC_TRIM_THRESHOLD_ 5693
setenv MALLOC_MMAP_THRESHOLD_ 384

From my uderstanding the MALLOC_MMAP_THRESHOLD_ does most of the work here. This parameter states at how many byes Malloc uses from the heap before using MMAP. By default this is huge: 128KB. This means gem5 uses as lot of the heap. By using setting this to 384 gem5/malloc is forced into using MMAP more which is more efficient (Langdon notes any value between 256 and 512 is pretty much the same).

Given these are environmental variables I'm not sure there's a safe way to distribute these configurations as part of our build system (at least I don't think there is). We could add a note in a README somewhere. I write this discussion to disseminate this information more than anything else

Note: I think if you use TCMalloc with your gem5 build you get a lot of this optimization done for you (it's my understanding TCMalloc is more clever with its use of MMAP). However, if we can achieve the same just through tweaking regular Malloc parameters then i feel that's better. It's my understanding many systems don't support TCMalloc.