Thread safety issue when loading dynamic modules on Linux

[jackcmay]

Observed while using crate libloading

Given a multi-threaded test that loads a dynamic module, calls a function, and then unloads the module in a loop, the expected behavior is for the module to be loaded/unloaded and it's init/fini called respectively everytime. Like so:

test start
init
dynamic function
fini
init
dynamic function
fini
test end

The expected behavior is observed when running on MacOS but not on Linux.

On Linux we observe the module's init/deinit function being called at the beginning and end of the test but also sporadically (but always in order) throughout the test. This hints that modules are not being unloaded each time (and instead shared amongst threads).

test start
init
dynamic function
.....
fini
init
dynamic function
fini
init
dynamic function
...
fini
test end

In addition to the behavior difference, on Linux this behavior also mostly results in a segmentation fault at some point in the program's execution.

No workaround to the behavior differences is known, but there is a workaround to the segmentation fault. Using RTLD_NODELETE when loading the module forces the loader to not unload the module until the process is killed (init called at beginning of test and fini at the very end).

solana/src/dynamic_program.rs

Line 78 in dffb77b

libloading::os::unix::Library::open(Some(path), libc::RTLD_NODELETE | libc::RTLD_NOW)

The following repository will recreate the issue: https://github.com/jackcmay/dynamic_module_segfault

This issue is not to be confused with another loading issue on Linux having to do with TLS:

• Unloading a Rust dylib with TLS used segfaults on OSX rust-lang/rust#28794
• TLS destructors are not run on Library::drop resulting in illegal instruction on OS X nagisa/rust_libloading#5
• Dynamic library, OSX, Segmentation fault rust-lang/rust#38370
• SEGFAULT sometimes occurs when testing libloading code nagisa/rust_libloading#41

[jackcmay]

Solana uses dynamic loading for native programs and native programs are loaded once and kept loaded so this issue does not apply