VM crashes on shutdown

[jimbeveridge]

We are seeing a flakey crash on Dart VM shutdown when our Mojo app exits. The gdb stack backtrace is shown below. This can't be reduced to a simple repro case, so I'm not sure how to proceed.

#0  base::debug::(anonymous namespace)::StackDumpSignalHandler (
    signal=<error reading variable: Cannot access memory at address 0xffffffffffffffd4>, 
    info=<error reading variable: Cannot access memory at address 0xffffffffffffffc8>, 
    void_context=<error reading variable: Cannot access memory at address 0xffffffffffffffc0>)
    at ../../base/debug/stack_trace_posix.cc:208
#1  <signal handler called>
#2  0x00007f680401bb6e in dart::Monitor::Exit (this=<optimised out>) at ../../dart/runtime/vm/os_thread_linux.cc:306
#3  0x00007f68040e4623 in dart::ThreadPool::Worker::Loop (this=<optimised out>) at ../../dart/runtime/vm/thread_pool.cc:277
#4  0x00007f68040e4594 in dart::ThreadPool::Worker::Main (args=<optimised out>) at ../../dart/runtime/vm/thread_pool.cc:320
#5  0x00007f680401b2fb in dart::ThreadStart (data_ptr=<optimised out>) at ../../dart/runtime/vm/os_thread_linux.cc:86
#6  0x00007f6808466182 in start_thread (arg=0x7f680201b700) at pthread_create.c:312
#7  0x00007f680819347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

[kodandersson]

We do suspect there are some issues surrounding shutdown. It would be great to get a core dump from such a crash. Could you please run the app under "ulimit -c unlimited" and send me any core that it dumps, and the binary?

[kodandersson]

This looks like failure to cleanly shut down threads in the thread pool.

In the core file, "Dart::thread_pool_" is NULL, and we currently never clear that field once it's set. Also another thread's backtrace suggests the entire dart lib is being unmapped, which would explain the unexpected zero memory:

This seems to be related to work in progress by @zanderso here:
https://codereview.chromium.org/1177153005/

(gdb) bt
#0 0x00007f68092c1687 in munmap () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007f68092c0ddd in _dl_unmap (map=map@entry=0x14266387b300) at ../sysdeps/x86_64/tlsdesc.c:139
#2 0x00007f68092bdbf7 in _dl_close_worker (map=map@entry=0x14266387b300) at dl-close.c:639
#3 0x00007f68092be48c in _dl_close (_map=0x14266387b300) at dl-close.c:773
#4 0x00007f68092b7ff4 in _dl_catch_error (objname=0x142663846a90, errstring=0x142663846a98, mallocedp=0x142663846a88,
operate=0x7f68090a50e0 <dlclose_doit>, args=0x14266387b300) at dl-error.c:187
#5 0x00007f68090a562d in _dlerror_run (operate=operate@entry=0x7f68090a50e0 <dlclose_doit>, args=0x14266387b300) at dlerror.c:163
#6 0x00007f68090a510f in __dlclose (handle=) at dlclose.c:46
#7 0x00000000004a2d2b in base::UnloadNativeLibrary (library=0x14266387b300) at ../../base/native_library_posix.cc:39
#8 0x00000000004c223a in base::ScopedNativeLibrary::~ScopedNativeLibrary (this=0x1426638414e0) at ../../base/scoped_native_library.cc:22
#9 0x0000000000427cbb in shell::InProcessNativeRunner::~InProcessNativeRunner (this=0x1426638414b0)
at ../../shell/in_process_native_runner.cc:29
#10 0x0000000000427d79 in shell::InProcessNativeRunner::~InProcessNativeRunner (this=0x1426638414b0)
at ../../shell/in_process_native_runner.cc:20
#11 0x0000000000c23c4b in STLDeleteContainerPointers<__gnu_debug::_Safe_iterator<__gnu_cxx::__normal_iterator<shell::NativeRunner**, std::__cxx1998::vector<shell::NativeRunner*, std::allocatorshell::NativeRunner* > >, std::__debug::vector<shell::NativeRunner*, std::allocatorshell::NativeRunner* > > > (begin=..., end=...) at ../../base/stl_util.h:44
#12 0x0000000000c23bb3 in STLDeleteElements<std::__debug::vector<shell::NativeRunner*, std::allocatorshell::NativeRunner* > > (
container=0x7fff238fbd88) at ../../base/stl_util.h:148
#13 0x0000000000c23b25 in ScopedVectorshell::NativeRunner::clear (this=0x7fff238fbd88) at ../../base/memory/scoped_vector.h:99
#14 0x0000000000c1e5d9 in ScopedVectorshell::NativeRunner::~ScopedVector (this=0x7fff238fbd88) at ../../base/memory/scoped_vector.h:38
#15 0x0000000000c19ea7 in shell::ApplicationManager::~ApplicationManager (this=0x7fff238fbb18)
at ../../shell/application_manager/application_manager.cc:109
#16 0x000000000041dba0 in shell::Context::~Context (this=0x7fff238fbb00) at ../../shell/context.cc:233
#17 0x000000000040c70d in main (argc=7, argv=0x7fff238fc3e8) at ../../shell/desktop/main.cc:130

[zanderso]

Clean shutdown has been turned on for a long time, and the upstream problem seems to be gone as well.