Deadlock on Windows when unloading libraries in several threads

[aseyboldt]

While working on the rust backend we noticed problems that appear when stan modules are loaded and unloaded in separate threads.

On windows we see a deadlock under the following conditions:

Start three threads. Load stan module A in the first thread, while the other threads wait. Load module B in the second thread. Then, unload module A in the first thread and exit the thread. And last, try to load module C in the third thread. This will deadlock. From some debugging it seems that the second thread actually died somewhere while exiting.
This exact behavior seems to depend on the compiler or tbb version, because if we compile the stan model with clang or msvc instead of mingw and a recent TBB, we do not see the same issue (see #105 (comment))

This might be related to TBB, we found some issues that look at bit similar to this on the TBB bugtracker (#105 (comment)).

A reproducer using the rust backend can be found here: https://github.com/aseyboldt/bridgestan/blob/main/rust/tests/create_models.rs#L236

There also seem to be some sporadic issues involving library unloading on linux as well however. With compilation settings O=0 and enabled library unloading I observed a couple of sporadic segfaults when I ran the rust test. In a debugger it looked like the regular expression here was not correctly initialized. I don't know for sure that this is related to unloading the libraries, but I've never seen that issue without it. And I think that since c++11 initialization of static variables should actually be thread safe.

(cc @WardBrian Feel free to add or modify the summary as you like)

[aseyboldt]

I guess if I had checked the libloading issues a bit more thoroughly I might have saved myself quite a bit of time...

nagisa/rust_libloading#41
nagisa/rust_libloading#5