pantsd nailgunerror segfault

[ity]

When attempting to run pants commands in our local repo, I encounter this error:

$ ./pants test $target
CRITICAL] 
CRITICAL] lost active connection to pantsd!
timestamp: 2018-10-31T16:18:53.566498
Exception caught: (pants.bin.remote_pants_runner.Terminated)
  File "/Users/ity/.cache/pants/pex/bin/pants.pex/1.11.0rc0-f9a61c0c/pants.pex/.bootstrap/_pex/pex.py", line 351, in execute
    self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/Users/ity/.cache/pants/pex/bin/pants.pex/1.11.0rc0-f9a61c0c/pants.pex/.bootstrap/_pex/pex.py", line 281, in _wrap_coverage
    runner(*args)
  File "/Users/ity/.cache/pants/pex/bin/pants.pex/1.11.0rc0-f9a61c0c/pants.pex/.bootstrap/_pex/pex.py", line 313, in _wrap_profiling
    runner(*args)

  File "/Users/ity/.cache/pants/pex/bin/pants.pex/1.11.0rc0-f9a61c0c/pants.pex/.bootstrap/_pex/pex.py", line 394, in _execute

    return self.execute_entry(self._pex_info.entry_point)

  File "/Users/ity/.cache/pants/pex/bin/pants.pex/1.11.0rc0-f9a61c0c/pants.pex/.bootstrap/_pex/pex.py", line 505, in execute_entry

    return runner(entry_point)

  File "/Users/ity/.cache/pants/pex/bin/pants.pex/1.11.0rc0-f9a61c0c/pants.pex/.bootstrap/_pex/pex.py", line 527, in execute_pkg_resources

    return runner()

  File "/Users/ity/workspace/source/.pex/install/pantsbuild.pants-1.11.0rc0+f9a61c0c-cp27-none-macosx_10_11_x86_64.whl.545e36657048c66e803f821fe09b152f4e558d92/pantsbuild.pants-1.11.0rc0+f9a61c0c-cp27-none-macosx_10_11_x86_64.whl/pants/bin/pants_loader.py", line 71, in main

    PantsLoader.run()

  File "/Users/ity/workspace/source/.pex/install/pantsbuild.pants-1.11.0rc0+f9a61c0c-cp27-none-macosx_10_11_x86_64.whl.545e36657048c66e803f821fe09b152f4e558d92/pantsbuild.pants-1.11.0rc0+f9a61c0c-cp27-none-macosx_10_11_x86_64.whl/pants/bin/pants_loader.py", line 67, in run

    cls.load_and_execute(entrypoint)

  File "/Users/ity/workspace/source/.pex/install/pantsbuild.pants-1.11.0rc0+f9a61c0c-cp27-none-macosx_10_11_x86_64.whl.545e36657048c66e803f821fe09b152f4e558d92/pantsbuild.pants-1.11.0rc0+f9a61c0c-cp27-none-macosx_10_11_x86_64.whl/pants/bin/pants_loader.py", line 60, in load_and_execute

    entrypoint_main()


  File "/Users/ity/workspace/source/.pex/install/pantsbuild.pants-1.11.0rc0+f9a61c0c-cp27-none-macosx_10_11_x86_64.whl.545e36657048c66e803f821fe09b152f4e558d92/pantsbuild.pants-1.11.0rc0+f9a61c0c-cp27-none-macosx_10_11_x86_64.whl/pants/bin/pants_exe.py", line 39, in main

    PantsRunner(exiter, start_time=start_time).run()

  File "/Users/ity/workspace/source/.pex/install/pantsbuild.pants-1.11.0rc0+f9a61c0c-cp27-none-macosx_10_11_x86_64.whl.545e36657048c66e803f821fe09b152f4e558d92/pantsbuild.pants-1.11.0rc0+f9a61c0c-cp27-none-macosx_10_11_x86_64.whl/pants/bin/pants_runner.py", line 48, in run

This resolves as soon as I turn the daemon off. The exceptions.log and PEX-INFO is attached. Also, important to note that this currently occurs only on osx 10.12. osx 10.13 and above do not exhibit the issue afaik.

exceptions (1).log
PEX-INFO (1).txt

[jsirois]

There is just not enough to go on here. The only fishy looking thing is:

Thread 0x00007000020f0000 (most recent call first):
  File "/Users/ity/.pex/install/setuptools-40.4.3-py2.py3-none-any.whl.2a195c3ab69b2327784ba2ff2f549025346caab5/setuptools-40.4.3-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 1439 in _parseNoCache
  File "/Users/ity/.pex/install/setuptools-40.4.3-py2.py3-none-any.whl.2a195c3ab69b2327784ba2ff2f549025346caab5/setuptools-40.4.3-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 1404 in _parseNoCache
  File "/Users/ity/.pex/install/setuptools-40.4.3-py2.py3-none-any.whl.2a195c3ab69b2327784ba2ff2f549025346caab5/setuptools-40.4.3-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 3400 in _parseNoCache
  File "/Users/ity/.pex/install/setuptools-40.4.3-py2.py3-none-any.whl.2a195c3ab69b2327784ba2ff2f549025346caab5/setuptools-40.4.3-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 1404 in _parseNoCache
  File "/Users/ity/.pex/install/setuptools-40.4.3-py2.py3-none-any.whl.2a195c3ab69b2327784ba2ff2f549025346caab5/setuptools-40.4.3-py2.py3-none-any.whl/pkg_resources/_vendor/pyparsing.py", line 1404 in _parseNoCache

And its only fishy because it has the last two stack lines being the same - which doesn't make sense generally or even specifically looking at line 1404 of pkg_resources/_vendor/pyparsing.py in setuptools-40.4.3.

I'd say there is much more digging to do.

[ity]

lldb backtrace on the stopped thread:

(lldb) thread backtrace
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
* thread #7, name = 'tokio-runtime-worker-1', stop reason = EXC_BAD_ACCESS (code=1, address=0x110)
  * frame #0: 0x00007fffb48085c0 libdispatch.dylib`_dispatch_root_queue_push_override_stealer + 208
    frame #1: 0x00007fffb4806d60 libdispatch.dylib`_dispatch_queue_class_wakeup_with_override + 420
    frame #2: 0x00007fffb47fb564 libdispatch.dylib`_dispatch_barrier_sync_f_slow + 371
    frame #3: 0x00007fffb47c9f45 libcorecrypto.dylib`ccrng_cryptographic_generate + 326
    frame #4: 0x00007fffb47559e2 libcommonCrypto.dylib`CCRandomCopyBytes + 100
    frame #5: 0x000000010f2eaddb native_engine.so`_$LT$rand..rngs..os..OsRng$u20$as$u20$rand_core..RngCore$GT$::try_fill_bytes::h8099276f93fdbbd9 [inlined] _$LT$rand..rngs..os..imp..OsRng$u20$as$u20$rand..rngs..os..OsRngImpl$GT$::fill_chunk::h8f5f79f3912c5c55(self=<unavailable>, dest=<unavailable>) at os.rs:806 [opt]
    frame #6: 0x000000010f2eadd1 native_engine.so`_$LT$rand..rngs..os..OsRng$u20$as$u20$rand_core..RngCore$GT$::try_fill_bytes::h8099276f93fdbbd9(self=<unavailable>, dest=<unavailable>) at os.rs:213 [opt]
    frame #7: 0x000000010f2e9d78 native_engine.so`_$LT$rand..rngs..entropy..EntropyRng$u20$as$u20$rand_core..RngCore$GT$::try_fill_bytes::h703baee7e6e70666 at entropy.rs:234 [opt]
    frame #8: 0x000000010f2e9d6d native_engine.so`_$LT$rand..rngs..entropy..EntropyRng$u20$as$u20$rand_core..RngCore$GT$::try_fill_bytes::h703baee7e6e70666(self=0x000070000a5b9a08, dest=(data_ptr = <no value available>, length = 32)) at entropy.rs:105 [opt]
    frame #9: 0x000000010f2eafcb native_engine.so`rand::rngs::thread::THREAD_RNG_KEY::__init::h773dab754d0f504d [inlined] _$LT$$RF$$u27$a$u20$mut$u20$R$u20$as$u20$rand_core..RngCore$GT$::try_fill_bytes::he444b691e0249de1(dest=(data_ptr = <no value available>, length = 32)) at lib.rs:361 [opt]
    frame #10: 0x000000010f2eafc1 native_engine.so`rand::rngs::thread::THREAD_RNG_KEY::__init::h773dab754d0f504d at lib.rs:335 [opt]
    frame #11: 0x000000010f2eaf9b native_engine.so`rand::rngs::thread::THREAD_RNG_KEY::__init::h773dab754d0f504d at thread.rs:83 [opt]
    frame #12: 0x000000010f2eacdd native_engine.so`rand::rngs::thread::thread_rng::h57ea5228003b18db [inlined] _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::init::h964d87ec6adf18be(slot=<unavailable>) at local.rs:255 [opt]
    frame #13: 0x000000010f2eacd8 native_engine.so`rand::rngs::thread::thread_rng::h57ea5228003b18db at local.rs:296 [opt]
    frame #14: 0x000000010f2eac8f native_engine.so`rand::rngs::thread::thread_rng::h57ea5228003b18db [inlined] _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::h423542e1711d3a68 at local.rs:248 [opt]
    frame #15: 0x000000010f2eac8f native_engine.so`rand::rngs::thread::thread_rng::h57ea5228003b18db at thread.rs:101 [opt]
    frame #16: 0x000000010f1e28ce native_engine.so`tokio_threadpool::pool::Pool::rand_usize::h6eafa2b326c68456 [inlined] rand::random::hfc2bb8592becb723 at lib.rs:1010 [opt]
    frame #17: 0x000000010f1e28c9 native_engine.so`tokio_threadpool::pool::Pool::rand_usize::h6eafa2b326c68456 [inlined] tokio_threadpool::pool::Pool::rand_usize::RNG::__init::h3b80290e26ba3daa at mod.rs:518 [opt]
    frame #18: 0x000000010f1e28c9 native_engine.so`tokio_threadpool::pool::Pool::rand_usize::h6eafa2b326c68456 [inlined] _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::init::he0ce635efae8cd07 at local.rs:255 [opt]
    frame #19: 0x000000010f1e28c9 native_engine.so`tokio_threadpool::pool::Pool::rand_usize::h6eafa2b326c68456 [inlined] _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::h65e3be47bbedbd92 at local.rs:296 [opt]
    frame #20: 0x000000010f1e28a4 native_engine.so`tokio_threadpool::pool::Pool::rand_usize::h6eafa2b326c68456 [inlined] _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::h930467e247a47377 at local.rs:248 [opt]
    frame #21: 0x000000010f1e28a4 native_engine.so`tokio_threadpool::pool::Pool::rand_usize::h6eafa2b326c68456(self=<unavailable>) at mod.rs:523 [opt]
    frame #22: 0x000000010f1e33d9 native_engine.so`tokio_threadpool::worker::Worker::run::hb6bd565eba9f8cc7 at mod.rs:408 [opt]
    frame #23: 0x000000010f1e33cd native_engine.so`tokio_threadpool::worker::Worker::run::hb6bd565eba9f8cc7 [inlined] tokio_threadpool::worker::Worker::try_run_task::h3878379ff13c0506(self=<unavailable>, notify=<unavailable>) at mod.rs:303 [opt]
    frame #24: 0x000000010f1e3386 native_engine.so`tokio_threadpool::worker::Worker::run::hb6bd565eba9f8cc7(self=0x000070000a5b9cf0) at mod.rs:240 [opt]
    frame #25: 0x000000010f1da23f native_engine.so`tokio::runtime::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::h9d8e9a3ceff0c87c [inlined] tokio::runtime::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h8a87126cb13ffe8a at builder.rs:125 [opt]
    frame #26: 0x000000010f1da236 native_engine.so`tokio::runtime::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::h9d8e9a3ceff0c87c [inlined] tokio_timer::timer::handle::with_default::_$u7b$$u7b$closure$u7d$$u7d$::h75e4e923a8a84bf2(current=<unavailable>) at handle.rs:94 [opt]
    frame #27: 0x000000010f1da1fd native_engine.so`tokio::runtime::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::h9d8e9a3ceff0c87c at local.rs:294 [opt]
    frame #28: 0x000000010f1da18b native_engine.so`tokio::runtime::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::h9d8e9a3ceff0c87c [inlined] _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::hdde6a023c1b96eeb(f=<unavailable>) at local.rs:248 [opt]
    frame #29: 0x000000010f1da18b native_engine.so`tokio::runtime::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::h9d8e9a3ceff0c87c at handle.rs:81 [opt]
    frame #30: 0x000000010f1da18b native_engine.so`tokio::runtime::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::h9d8e9a3ceff0c87c [inlined] tokio::runtime::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h748e53947508e13e(enter=<unavailable>) at builder.rs:124 [opt]
    frame #31: 0x000000010f1da18b native_engine.so`tokio::runtime::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::h9d8e9a3ceff0c87c at clock.rs:136 [opt]
    frame #32: 0x000000010f1da17c native_engine.so`tokio::runtime::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::h9d8e9a3ceff0c87c at local.rs:294 [opt]
    frame #33: 0x000000010f1da137 native_engine.so`tokio::runtime::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::h9d8e9a3ceff0c87c [inlined] _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::hd087f79099a19030(f=<unavailable>) at local.rs:248 [opt]
    frame #34: 0x000000010f1da137 native_engine.so`tokio::runtime::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::h9d8e9a3ceff0c87c [inlined] tokio_timer::clock::clock::with_default::h293730bbb3901e00(clock=<unavailable>, enter=<unavailable>, f=<unavailable>) at clock.rs:119 [opt]
    frame #35: 0x000000010f1da137 native_engine.so`tokio::runtime::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::h9d8e9a3ceff0c87c [inlined] tokio::runtime::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::hcf62cad402c783c1(enter=<unavailable>) at builder.rs:123 [opt]
    frame #36: 0x000000010f1da137 native_engine.so`tokio::runtime::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::h9d8e9a3ceff0c87c [inlined] tokio_reactor::with_default::_$u7b$$u7b$closure$u7d$$u7d$::hf88f635cfe6d203b(current=<unavailable>) at lib.rs:232 [opt]
    frame #37: 0x000000010f1da0f8 native_engine.so`tokio::runtime::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::h9d8e9a3ceff0c87c at local.rs:294 [opt]
    frame #38: 0x000000010f1da086 native_engine.so`tokio::runtime::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::h9d8e9a3ceff0c87c [inlined] _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::h1e1b541c7f507212(f=<unavailable>) at local.rs:248 [opt]
    frame #39: 0x000000010f1da086 native_engine.so`tokio::runtime::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::h9d8e9a3ceff0c87c [inlined] tokio_reactor::with_default::he7dc98bbb445aeec(handle=<unavailable>, f=closure @ 0x00007f9bd90a80d0) at lib.rs:215 [opt]
    frame #40: 0x000000010f1da086 native_engine.so`tokio::runtime::builder::Builder::build::_$u7b$$u7b$closure$u7d$$u7d$::h9d8e9a3ceff0c87c(w=<unavailable>, enter=<unavailable>) at builder.rs:122 [opt]
    frame #41: 0x000000010f1dc180 native_engine.so`std::sys_common::backtrace::__rust_begin_short_backtrace::h58ab36c3e20877d8 [inlined] tokio_threadpool::callback::Callback::call::hcc71896bc5f3378c(worker=0x00007fbca617ec40, enter=0x0000000000000008) at callback.rs:21 [opt]
    frame #42: 0x000000010f1dc166 native_engine.so`std::sys_common::backtrace::__rust_begin_short_backtrace::h58ab36c3e20877d8 at mod.rs:122 [opt]
    frame #43: 0x000000010f1dc153 native_engine.so`std::sys_common::backtrace::__rust_begin_short_backtrace::h58ab36c3e20877d8 at global.rs:173 [opt]
    frame #44: 0x000000010f1dc113 native_engine.so`std::sys_common::backtrace::__rust_begin_short_backtrace::h58ab36c3e20877d8 at local.rs:294 [opt]
    frame #45: 0x000000010f1dc0f0 native_engine.so`std::sys_common::backtrace::__rust_begin_short_backtrace::h58ab36c3e20877d8 [inlined] _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::h1d9f56d2343c2631 at local.rs:248 [opt]
    frame #46: 0x000000010f1dc0f0 native_engine.so`std::sys_common::backtrace::__rust_begin_short_backtrace::h58ab36c3e20877d8 [inlined] tokio_executor::global::with_default::h76a0165a9be148f2(executor=0x00007fbca617ec40) at global.rs:143 [opt]
    frame #47: 0x000000010f1dc0f0 native_engine.so`std::sys_common::backtrace::__rust_begin_short_backtrace::h58ab36c3e20877d8 at mod.rs:120 [opt]
    frame #48: 0x000000010f1dc096 native_engine.so`std::sys_common::backtrace::__rust_begin_short_backtrace::h58ab36c3e20877d8 [inlined] _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::h8fee7ae24246f55a at local.rs:294 [opt]
    frame #49: 0x000000010f1dc062 native_engine.so`std::sys_common::backtrace::__rust_begin_short_backtrace::h58ab36c3e20877d8 [inlined] _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::h75f2475d58df0ddd at local.rs:248 [opt]
    frame #50: 0x000000010f1dc062 native_engine.so`std::sys_common::backtrace::__rust_begin_short_backtrace::h58ab36c3e20877d8 [inlined] tokio_threadpool::worker::Worker::do_run::h49bf13a4a8e5053d at mod.rs:111 [opt]
    frame #51: 0x000000010f1dc062 native_engine.so`std::sys_common::backtrace::__rust_begin_short_backtrace::h58ab36c3e20877d8 at mod.rs:413 [opt]
    frame #52: 0x000000010f1dbfbb native_engine.so`std::sys_common::backtrace::__rust_begin_short_backtrace::h58ab36c3e20877d8(f=<unavailable>) at backtrace.rs:136 [opt]
    frame #53: 0x000000010f1dc648 native_engine.so`std::panicking::try::do_call::h281e5da3036a8c40 [inlined] std::thread::Builder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::hf704d5da83c8c77e at mod.rs:409 [opt]
    frame #54: 0x000000010f1dc628 native_engine.so`std::panicking::try::do_call::h281e5da3036a8c40 [inlined] _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h2dd46ddc7dac29e4 at panic.rs:313 [opt]
    frame #55: 0x000000010f1dc628 native_engine.so`std::panicking::try::do_call::h281e5da3036a8c40(data=<unavailable>) at panicking.rs:310 [opt]
    frame #56: 0x000000010f5bf42f native_engine.so`__rust_maybe_catch_panic at lib.rs:105 [opt]
    frame #57: 0x000000010f1de9ba native_engine.so`_$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::ha49e5f3c24d4063e at panicking.rs:289 [opt]
    frame #58: 0x000000010f1de97a native_engine.so`_$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::ha49e5f3c24d4063e [inlined] std::panic::catch_unwind::h739e835a53e7ec64 at panic.rs:392 [opt]
    frame #59: 0x000000010f1de97a native_engine.so`_$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::ha49e5f3c24d4063e [inlined] std::thread::Builder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::h4aeecb2152df4f76 at mod.rs:408 [opt]
    frame #60: 0x000000010f1de94f native_engine.so`_$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::ha49e5f3c24d4063e(self=0x00007fbca60343c0, args=<unavailable>) at boxed.rs:642 [opt]
    frame #61: 0x000000010f5b2928 native_engine.so`std::sys_common::thread::start_thread::hffea5e9f9f11b206 [inlined] _$LT$alloc..boxed..Box$LT$$LP$dyn$u20$alloc..boxed..FnBox$LT$A$C$$u20$Output$u3d$R$GT$$u20$$u2b$$u20$$u27$a$RP$$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::h96fdc6e64ca590c2 at boxed.rs:652 [opt]
    frame #62: 0x000000010f5b2925 native_engine.so`std::sys_common::thread::start_thread::hffea5e9f9f11b206 at thread.rs:24 [opt]
    frame #63: 0x000000010f59c409 native_engine.so`std::sys::unix::thread::Thread::new::thread_start::hce603054d9b5b6fc at thread.rs:90 [opt]
    frame #64: 0x00007fffb4a3f93b libsystem_pthread.dylib`_pthread_body + 180
    frame #65: 0x00007fffb4a3f887 libsystem_pthread.dylib`_pthread_start + 286
    frame #66: 0x00007fffb4a3f08d libsystem_pthread.dylib`thread_start + 13

[illicitonion]

EXC_BAD_ACCESS in libdispatch.dylib suggests to me that this is a fork-safety issue. I believe that libdispatch starts up background threads, which makes it unsafe to use when we fork and don't exec.

[stuhood]

#7323 might be a dupe of this one.

[Eric-Arellano]

This is being encountered very frequently in the setup repo's CI, to the point where the shard almost always fails. See here, here, and here.

It appears to be hitting us when using both Pants 1.14.0 and 1.15.3.

Notably, this only impacts OSX 10.12, not 10.11 or 10.13.

[stuhood]

Closed via #7596!