jemalloc_pool should not require user to link with jemalloc

[lplewa]

Currently user needs to pass -ljemalloc linker flag if uses jemalloc pool (jemalloc_pool.a)

This is an issue as jemalloc exports malloc, realloc, free symbols, so actually liking with jemalloc overrides system functions.
That is annoying for the user, who don't want to use jmalloc instead of glibc functions.

This is also a problem for proxy lib's threshold #883 as dlsym(RTLD_NEXT, "malloc"); will return malloc@jemalloc not malloc@glibc. See also #894.

to overcome this issue we can:

• use dlopen, and dlsym in jemalloc_pool
• link statically with jemaloc in jemalloc_pool

Not sure what is a better approach.

[vinser52]

Just to note here what I mentioned on the UMF Technical meeting:

link statically with jemalloc in jemalloc_pool

It would mean that UMF is hardcoded to use a particular version of jemalloc, In the past we heard a lot of concerns from Memkind customers/questions regarding how to force Memkind to use different versions of jemalloc. With the current design of UMF users can just upgrade jemalloc to the required version. But if we link statically then UMF should be rebuilt to use a different version of jemalloc.

[bratpiorka]

I'm for dynamic linking

[lplewa]

statically link allows use to use CTL macros from jemaloc header - but by looking on the header, there is not a huge amount of macros which looks useful for us, so we should be fine.

@pbalcer can to show any examples of jemalloc macros which might be a limiting factor for us if we decide to dynamic link jemaloc with umf.

[lplewa]

I talked with Piotr, and i convinced him that we can go with dynamic link to fix this issue

[ldorau]

@bratpiorka @vinser52 @lplewa @pbalcer there is an issue with dlopen(libjemalloc.so):

Dynamically loading the jemalloc library (using dlopen(libjemalloc.so)) requires rebuilding it from sources with the "--disable-initial-exec-tls" option, because of the following issue: jemalloc/jemalloc#1237

I get the following error now:

[FATAL UMF] utils_open_library: dlopen(libjemalloc.so.2) failed with error: /usr/lib/x86_64-linux-gnu/libjemalloc.so.2: cannot allocate memory in static TLS block

See: #919

[bratpiorka]

@vinser52 @lplewa @pbalcer what do you think of the following combo:
when jemalloc is required: fetch jemalloc from github + build with required flags + add an option to use jemalloc the user-provided path (this is a user's responsibility to build it properly) + load so dynamically + ship .so with UMF for oneAPI package

[lplewa]

@vinser52 @lplewa @pbalcer what do you think of the following combo: when jemalloc is required: fetch jemalloc from github + build with required flags + add an option to use jemalloc the user-provided path (this is a user's responsibility to build it properly) + load so dynamically + ship .so with UMF for oneAPI package

Fetch jemalloc with cmake option to use different version, and link statically. You approach have an issue that on make install of umf we have to install our version of jemalloc.so, what is not a great idea.

[pbalcer]

The main argument for using dlopen was that it'd be possible for users to supplement their own jemalloc from the OS. That is no longer the case, and would require us to explain to users that they need to have a specially built jemalloc with the tls flag.
At this point I think the cons outweigh the pros of using dlopen. I suggest we just do what memkind does and build/link with jemalloc statically (setting --disable-initial-exec-tls to fix any potential issues with dlopening umf itself).

[vinser52]

The main reason to link dynamically was the following: we remember that Memkind users asked from time to time that they would like to use different version of jemalloc with Memkind. If we cannot reach it we UMF than we can just link statically with particular version of jemalloc.

Regarding dlopen approach, what if we will create a proxy lib ( e.g. jemalloc_proxy, name could be different :) ) that will have a hard dependency to jemalloc.so. umf will try to dlopen the jemalloc_proxy. Since jemalloc_proxy depends on libjemalloc.so the dlopen will successes if libjemalloc.so is available on the system or fail otherwise.

[vinser52]

The main argument for using dlopen was that it'd be possible for users to supplement their own jemalloc from the OS. That is no longer the case, and would require us to explain to users that they need to have a specially built jemalloc with the tls flag. At this point I think the cons outweigh the pros of using dlopen. I suggest we just do what memkind does and build/link with jemalloc statically (setting --disable-initial-exec-tls to fix any potential issues with dlopening umf itself).

@pbalcer explains almost the same :). I agree that if we are not able to allow users to choose the version of jemalloc than we should just link statically. But we can try the idea that I suggested in my previous comment.

[bratpiorka]

I would still like to provide the user with a way to rebuild the UMF with the jemalloc sources specified by the user. Is this possible?

[lplewa]

I would still like to provide the user with a way to rebuild the UMF with the jemalloc sources specified by the user. Is this possible?

Yeah - we will have a GH link with commit hash - in our cmake - user can just change. We can make also cmake variable to override it, so there is no need to change our files, but i prefer to not do so - we can add it always latter on, if someone will ask us. (we have tooo many variables in our cmake, so we should try to not add new if it is not needed)

[vinser52]

I would still like to provide the user with a way to rebuild the UMF with the jemalloc sources specified by the user. Is this possible?

Yeah - we will have a GH link with commit hash - in our cmake - user can just change. We can make also cmake variable to override it, so there is no need to change our files, but i prefer to not do so - we can add it always latter on, if someone will ask us. (we have tooo many variables in our cmake, so we should try to not add new if it is not needed)

I think @bratpiorka wanted an ability to load jemalloc dynamically if the user builds it from the source with --disable-initial-exec-tls option.

[pbalcer]

Regarding dlopen approach, what if we will create a proxy lib ( e.g. jemalloc_proxy, name could be different :) ) that will have a hard dependency to jemalloc.so. umf will try to dlopen the jemalloc_proxy. Since jemalloc_proxy depends on libjemalloc.so the dlopen will successes if libjemalloc.so is available on the system or fail otherwise.

That unfortunately won't work. The problem is transitive, dlopen would now complain that the jemalloc_proxy library is using too much TLS.

[ldorau]

The fix is ready, please review: #941