librvvm: Dynamic unloading doesn't deinit

[LekKit]

The issue

• When using shared librvvm, unloading it leaves some global state initialized
• This may cause segfaults from thread workers kept running, leaks, etc

Steps to reproduce

• Run Retroarch with libretro port of RVVM (See Add libretro core #96), I.e. load the lib
• Do some IO workloads, causing thread workers to run
• Click close content, which unloads the shared lib
• Segfault

Investigation

• Global data is deinitialized via use of atexit(), which is fine for standalone VM but not for a lib

Workarounds

• Omitting unloading the lib
• Waiting for a while after VM destruction so that thread workers exit

Suggested fix

• Implement rvvm_at_deinit() callbacks instead of atexit()
• Expose rvvm_deinit() API that is called before unloading the lib
• Perhaps mark it with GNU attributes for automatic deinit

[LekKit]

Implemented call_at_deinit() and full_deinit(), and moved all the infrastructure to use it.
On GNU-compatible compilers (Clang, GCC, other LLVM-based) it should deinit automatically on library unload.

@iyzsong can you please verify it doesn't segfault anymore?

[ghost]

well, it still segfault. full_deinit and thread_workers_terminate got called, but the worker threads are still running when it returned. if i understand correctly, in current code a worker thread doesn't terminate until THREAD_MAX_WORKER_IDLE timeout, and thread_workers_terminate doesn't wait it now (should be done by pthread_join or atomic flag check)...

[LekKit]

if i understand correctly, in current code a worker thread doesn't terminate

Ah sorry, makes sense. My local threadpool implementation is different, so I should upstream it soon and that's gonna be it

[LekKit]

Should be fixed now, the threadpool is properly joined on deinit.

[ghost]

Well, first "Close Content" does deinit correctly, but then load a rvvm file and close again will segfault, i think that's a RetroArch issue..

[LekKit]

I wrote a simple program that explicitly loads librvvm, runs a VM an unloads it (all done in a loop).
It will segfault if the machine is still running (I.e. rvvm_pause_machine() or rvvm_free_machine() wasn't called). Otherwise it is fine.

UPD: introduced a deinit hook that reaps every running machine and prints a warn. I don't know any other way that could cause a segv for now.

UPD: need to rewrite eventloop a bit to be dlclose-safe

[LekKit]

Implemented proper eventloop termination in 7ef1adc. There are no places where thread_detach() is used anymore.
There are some serious race issues if you keep running machines around till the actual deinit, but it is undefined from the API standpoint (And raises a warning if that ever happens).

Is it working OK now?

[ghost]

Yes, it's working, before and after commit 7ef1adc.

My previous issue was due to the second load by "History" will load a different rvvm_libretro.so than the latest built one... Its shutdown sequence rvvm_run_eventloop -> rvvm_free_machine -> full_deinit (auto called by DEINIT_ATTR) was correct (with a seperated vm_thread, which can be discarded now).

[LekKit]

Yes, it's working, before and after commit 7ef1adc

That commit fixes a race condition where the eventloop might run for a few spare moments after all machines have been shut down. I.e. without that commit there might be still a rare segv occurence, but now all is fixed.

Thanks for the PR!