Bug hard to hunt down under rr even with --chaos

[emilio]

I remember @rocallahan was interested on this kind of thing, so I'm opening an issue.

See: servo/ipc-channel#179

When I run the test on my machine, like:

$ true; while [ $? -eq 0 ];do RUST_BACKTRACE=1 ./target/release/deps/ipc_channel-6611382eff9c2dbb stress ; done

I manage to get something like:

failures:

---- test::stress_in_process stdout ----
	thread 'test::stress_in_process' panicked at 'recv: Io(Error { repr: Os { code: 104, message: "Connection reset by peer" } })', /checkout/src/libcore/result.rs:906:4
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace
             at /checkout/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::_print
             at /checkout/src/libstd/sys_common/backtrace.rs:69
   2: std::panicking::default_hook::{{closure}}
             at /checkout/src/libstd/sys_common/backtrace.rs:58
             at /checkout/src/libstd/panicking.rs:381
   3: std::panicking::default_hook
             at /checkout/src/libstd/panicking.rs:391
   4: std::panicking::rust_panic_with_hook
             at /checkout/src/libstd/panicking.rs:577
   5: std::panicking::begin_panic
             at /checkout/src/libstd/panicking.rs:538
   6: std::panicking::begin_panic_fmt
             at /checkout/src/libstd/panicking.rs:522
   7: rust_begin_unwind
             at /checkout/src/libstd/panicking.rs:498
   8: core::panicking::panic_fmt
             at /checkout/src/libcore/panicking.rs:71
   9: core::result::unwrap_failed
  10: ipc_channel::test::stress_in_process
  11: _ZN42_$LT$F$u20$as$u20$test..FnBox$LT$T$GT$$GT$8call_box17hee77675d0689c9f5E.llvm.313E8C1E
             at /checkout/src/libtest/lib.rs:1480
             at /checkout/src/libcore/ops/function.rs:223
             at /checkout/src/libtest/lib.rs:141
  12: __rust_maybe_catch_panic
             at /checkout/src/libpanic_unwind/lib.rs:99


failures:
    test::stress_in_process

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 69 filtered out

After 20 / 60 iterations consistently.

So the next logical step was to try hunting it with rr, using:

$ true; while [ $? -eq 0 ];do RUST_BACKTRACE=1 rr record --chaos ./target/debug/deps/ipc_channel-0a671e7e49351578 stress && rm -rf ~/.local/share/rr/ipc_channel-*; done

So far I haven't succeeded yet (it's been 90 iterations already, which take more than a minute when recording, both on debug and release).

Just filing to see if it rings a bell or you find it interesting, it's mostly a bunch of socketpair / sendmsg / recvmsg syscalls.

I'm running rr as of 244151f.

Also, something I've noticed about this test-case is that recording it under rr is much slower than not doing it even in release, but that's kinda expected given it's very syscall-intensive.

[rocallahan]

If you're doing a lot of ping-ponging between processes then rr's slowness is probably because we make context switches a lot more expensive, and that is very difficult to address.

[emilio]

It should only be across threads there (though it likely still applies).

[rocallahan]

OK, threads have the same issue.

Let me know when you've figured out the bug and maybe I can determine why chaos mode couldn't reproduce it, and fix it.