Unexpected aborts when exiting python after using rr.new_recording

[jleibs]

Both of these cases lead to:

terminate called without an active exception
Aborted (core dumped)

import rerun as rr

for _ in range(3):
    rec = rr.new_recording(application_id="test")
    mem = rec.memory_recording()
    rr.log("test", rr.Points3D([1, 2, 3]), recording=rec.inner)

import rerun as rr

rec = rr.new_recording(application_id="test")
rr.log("test", rr.Points3D([1, 2, 3]), recording=rec.inner)

However, a single memory recording appears to work fine without aborting:

rec = rr.new_recording(application_id="test")
rec.memory_recording()
rr.log("test", rr.Points3D([1, 2, 3]), recording=rec.inner)

[emilk]

It's very annoying that we don't get a stacktrace on SIGABRT

[teh-cmc]

Huh, interestingly both test cases behave properly at first and then get stuck indefinitely on Python 3.8, e.g.:

$ python main.py
DEV ENVIRONMENT DETECTED! Re-importing rerun from: /home/cmc/dev/rerun-io/rerun/rerun_py/rerun_sdk
[2023-12-01T16:21:03Z WARN  re_sdk::log_sink] Dropping data in MemorySink
[2023-12-01T16:21:03Z WARN  re_sdk::log_sink] Dropping data in MemorySink
[2023-12-01T16:21:03Z WARN  re_sdk::log_sink] Dropping data in MemorySink
[2023-12-01T16:21:03Z WARN  re_sdk::log_sink] Dropping data in MemorySink


# stuck, but no crash

^C

I do crash on Python 3.11.

[teh-cmc]

Backtrace from the core dump:

Text version:

#0  0x00007f2c088ac83c in ?? () from /usr/lib/libc.so.6
#1  0x00007f2c0885c668 in raise () from /usr/lib/libc.so.6
#2  0x00007f2c088444b8 in abort () from /usr/lib/libc.so.6
#3  0x00007f2c0549ca6f in __gnu_cxx::__verbose_terminate_handler () at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/vterminate.cc:95
#4  0x00007f2c054b011c in __cxxabiv1::__terminate (handler=<optimized out>) at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:48
#5  0x00007f2c054b0189 in std::terminate () at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:58
#6  0x00007f2c054af9cd in __cxxabiv1::__gxx_personality_v0 (version=<optimized out>, actions=10, exception_class=0, ue_header=<optimized out>, context=0x7f2baf3fc180) at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_personality.cc:673
#7  0x00007f2c079bb6f4 in _Unwind_ForcedUnwind_Phase2 (exc=exc@entry=0x7f2baf3fdd30, context=context@entry=0x7f2baf3fc180, frames_p=frames_p@entry=0x7f2baf3fc088) at /usr/src/debug/gcc/gcc/libgcc/unwind.inc:183
#8  0x00007f2c079bbe20 in _Unwind_ForcedUnwind (exc=0x7f2baf3fdd30, stop=0x7f2c088b2fa0, stop_argument=<optimized out>) at /usr/src/debug/gcc/gcc/libgcc/unwind.inc:218
#9  0x00007f2c088b3187 in ?? () from /usr/lib/libc.so.6
#10 0x00007f2c088abc3a in pthread_exit () from /usr/lib/libc.so.6
#11 0x00007f2c08b8a3e2 in PyThread_exit_thread () from /usr/lib/libpython3.11.so.1.0
#12 0x00007f2c08af153d in ?? () from /usr/lib/libpython3.11.so.1.0
#13 0x00007f2c08be0706 in PyEval_RestoreThread () from /usr/lib/libpython3.11.so.1.0
#14 0x00007f2c08cc90fa in PyGILState_Ensure () from /usr/lib/libpython3.11.so.1.0
#15 0x00007f2bb412b8ca in arrow::py::NumPyBuffer::~NumPyBuffer() () from /home/cmc/dev/rerun-io/rerun/venv310/lib/python3.11/site-packages/pyarrow/libarrow_python.so.1000
#16 0x00007f2bb411112e in std::_Sp_counted_ptr_inplace<arrow::ArrayData, std::allocator<arrow::ArrayData>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/cmc/dev/rerun-io/rerun/venv310/lib/python3.11/site-packages/pyarrow/libarrow_python.so.1000
#17 0x00007f2bb411108e in std::_Sp_counted_ptr_inplace<arrow::ArrayData, std::allocator<arrow::ArrayData>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/cmc/dev/rerun-io/rerun/venv310/lib/python3.11/site-packages/pyarrow/libarrow_python.so.1000
#18 0x00007f2bb0d6f6fa in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/cmc/dev/rerun-io/rerun/venv310/lib/python3.11/site-packages/pyarrow/libarrow.so.1000
#19 0x00007f2bb1831fec in arrow::(anonymous namespace)::ReleaseExportedArray(ArrowArray*) () from /home/cmc/dev/rerun-io/rerun/venv310/lib/python3.11/site-packages/pyarrow/libarrow.so.1000
#20 0x00007f2bb63c67e3 in arrow2::ffi::array::{impl#2}::drop (self=0x165a06) at src/ffi/array.rs:59
#21 core::ptr::drop_in_place<arrow2::ffi::generated::ArrowArray> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#22 alloc::sync::Arc<arrow2::ffi::generated::ArrowArray, alloc::alloc::Global>::drop_slow<arrow2::ffi::generated::ArrowArray, alloc::alloc::Global> (self=<optimized out>) at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/alloc/src/sync.rs:1751
#23 0x00007f2bb63c65bc in alloc::sync::{impl#33}::drop<arrow2::ffi::generated::ArrowArray, alloc::alloc::Global> (self=0x165a06) at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/alloc/src/sync.rs:2407
#24 core::ptr::drop_in_place<alloc::sync::Arc<arrow2::ffi::generated::ArrowArray, alloc::alloc::Global>> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#25 core::ptr::drop_in_place<arrow2::ffi::array::InternalArrowArray> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#26 core::ptr::drop_in_place<arrow2::buffer::BytesAllocator> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#27 core::ptr::drop_in_place<foreign_vec::Allocation<arrow2::buffer::BytesAllocator>> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#28 core::ptr::drop_in_place<foreign_vec::ForeignVec<arrow2::buffer::BytesAllocator, i32>> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#29 alloc::sync::Arc<foreign_vec::ForeignVec<arrow2::buffer::BytesAllocator, i32>, alloc::alloc::Global>::drop_slow<foreign_vec::ForeignVec<arrow2::buffer::BytesAllocator, i32>, alloc::alloc::Global> (self=<optimized out>) at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/alloc/src/sync.rs:1751
#30 0x00007f2bb63fb70c in alloc::sync::{impl#33}::drop<foreign_vec::ForeignVec<arrow2::buffer::BytesAllocator, f32>, alloc::alloc::Global> (self=0x165a06) at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/alloc/src/sync.rs:2407
#31 core::ptr::drop_in_place<alloc::sync::Arc<foreign_vec::ForeignVec<arrow2::buffer::BytesAllocator, f32>, alloc::alloc::Global>> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#32 core::ptr::drop_in_place<arrow2::buffer::immutable::Buffer<f32>> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#33 core::ptr::drop_in_place<arrow2::array::primitive::PrimitiveArray<f32>> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#34 0x00007f2bb63fb62b in core::ptr::drop_in_place<alloc::boxed::Box<dyn arrow2::array::Array, alloc::alloc::Global>> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#35 core::ptr::drop_in_place<arrow2::array::fixed_size_list::FixedSizeListArray> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#36 0x00007f2bb6204815 in core::ptr::drop_in_place<alloc::boxed::Box<dyn arrow2::array::Array, alloc::alloc::Global>> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#37 core::ptr::drop_in_place<arrow2::array::list::ListArray<i32>> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#38 0x00007f2bb62043f7 in core::ptr::drop_in_place<alloc::boxed::Box<dyn arrow2::array::Array, alloc::alloc::Global>> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#39 core::ptr::drop_in_place<[alloc::boxed::Box<dyn arrow2::array::Array, alloc::alloc::Global>]> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#40 alloc::vec::{impl#24}::drop<alloc::boxed::Box<dyn arrow2::array::Array, alloc::alloc::Global>, alloc::alloc::Global> (self=0x25200072448) at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/alloc/src/vec/mod.rs:3066
#41 core::ptr::drop_in_place<alloc::vec::Vec<alloc::boxed::Box<dyn arrow2::array::Array, alloc::alloc::Global>, alloc::alloc::Global>> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#42 core::ptr::drop_in_place<arrow2::chunk::Chunk<alloc::boxed::Box<dyn arrow2::array::Array, alloc::alloc::Global>>> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#43 core::ptr::drop_in_place<re_log_types::arrow_msg::ArrowMsg> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#44 0x00007f2bb5db8748 in core::ptr::drop_in_place<[re_log_types::LogMsg]> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#45 alloc::vec::{impl#24}::drop<re_log_types::LogMsg, alloc::alloc::Global> (self=0x251fc030d88) at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/alloc/src/vec/mod.rs:3066
#46 core::ptr::drop_in_place<alloc::vec::Vec<re_log_types::LogMsg, alloc::alloc::Global>> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#47 0x00007f2bb552b5aa in core::ptr::drop_in_place<alloc::boxed::Box<dyn re_sdk::log_sink::LogSink, alloc::alloc::Global>> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#48 re_sdk::recording_stream::forwarding_thread (info=..., sink=..., cmds_rx=..., tables=...) at crates/re_sdk/src/recording_stream.rs:998
#49 0x00007f2bb552ff15 in re_sdk::recording_stream::{impl#2}::new::{closure#0} () at crates/re_sdk/src/recording_stream.rs:599
#50 std::sys_common::backtrace::__rust_begin_short_backtrace<re_sdk::recording_stream::{impl#2}::new::{closure_env#0}, ()> (f=...) at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/sys_common/backtrace.rs:154
#51 0x00007f2bb5539eb7 in std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure#0}<re_sdk::recording_stream::{impl#2}::new::{closure_env#0}, ()> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/thread/mod.rs:529
#52 core::panic::unwind_safe::{impl#23}::call_once<(), std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<re_sdk::recording_stream::{impl#2}::new::{closure_env#0}, ()>> (self=<error reading variable: Cannot access memory at address 0x30>) at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/panic/unwind_safe.rs:271
#53 std::panicking::try::do_call<core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<re_sdk::recording_stream::{impl#2}::new::{closure_env#0}, ()>>, ()> (data=<optimized out>) at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:504
#54 std::panicking::try<(), core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<re_sdk::recording_stream::{impl#2}::new::{closure_env#0}, ()>>> (f=<error reading variable: Cannot access memory at address 0x30>) at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:468
#55 std::panic::catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<re_sdk::recording_stream::{impl#2}::new::{closure_env#0}, ()>>, ()> (f=<error reading variable: Cannot access memory at address 0x30>) at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panic.rs:142
#56 std::thread::{impl#0}::spawn_unchecked_::{closure#1}<re_sdk::recording_stream::{impl#2}::new::{closure_env#0}, ()> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/thread/mod.rs:528
#57 core::ops::function::FnOnce::call_once<std::thread::{impl#0}::spawn_unchecked_::{closure_env#1}<re_sdk::recording_stream::{impl#2}::new::{closure_env#0}, ()>, ()> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ops/function.rs:250
#58 0x00007f2bb66d6a05 in alloc::boxed::{impl#47}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:2007
#59 alloc::boxed::{impl#47}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:2007
#60 std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#61 0x00007f2c088aa9eb in ?? () from /usr/lib/libc.so.6
#62 0x00007f2c0892e7cc in ?? () from /usr/lib/libc.so.6

[teh-cmc]

from __future__ import annotations

import rerun as rr

# this works
rec = rr.new_recording(application_id="test")

# this crashes (coredump)
rec = rr.new_recording(application_id="test")
rr.log("test", rr.Points3D([1, 2, 3]), recording=rec.inner)

# this works
rec = rr.new_recording(application_id="test", make_default=True)
rr.log("test", rr.Points3D([1, 2, 3]), recording=rec.inner)

# this crashes (but with a segfault this time)
rec = rr.new_recording(application_id="test", make_thread_default=True)
rr.log("test", rr.Points3D([1, 2, 3]), recording=rec.inner)

# this works
rec = rr.new_recording(application_id="test") # this works
rr.set_global_data_recording(rec)
rr.log("test", rr.Points3D([1, 2, 3]), recording=rec.inner)

# this crashes (but with a segfault this time)
rec = rr.new_recording(application_id="test") # this works
rr.set_thread_local_data_recording(rec)
rr.log("test", rr.Points3D([1, 2, 3]), recording=rec.inner)

# this works!
rec = rr.new_recording(application_id="test", spawn=True)
rr.log("test", rr.Points3D([1, 2, 3]), recording=rec.inner)

# this works!
rec = rr.new_recording(application_id="test")
rr.connect(recording=rec)
rr.log("test", rr.Points3D([1, 2, 3]), recording=rec.inner)

# this works!
rec = rr.new_recording(application_id="test")
rr.memory_recording(recording=rec)
rr.log("test", rr.Points3D([1, 2, 3]), recording=rec.inner)

[teh-cmc]

Alright I see the problem and have a dirty fix working but gotta run!

[teh-cmc]

• Fixed by Python SDK: introduce deferred garbage collection queue #4583