ui test failures (segfaults and nested panics) since libbacktrace update

[nikic]

Since the libbacktrace update in #50955 I'm getting 150 failures in ui tests for my local rustc build. The configuration is default apart from enabling LLVM assertions. The system is Ubuntu 16.04 on x86_64. The issue occurs with 74d0939, but not with 4f99f37 (commit prior to #50955 merge).

The failures are either due to segfaults or panics while panicking.

Sample segfault:

#0  __GI___libc_free (mem=0x1f8) at malloc.c:2951
#1  0x00007ffff7a08a1a in core::ptr::drop_in_place::h1ca2358faa81af6e ()
   from /home/nikic/rust/build/x86_64-unknown-linux-gnu/stage1/bin/../lib/librustc_driver-34cfa81bde6fac8c.so
#2  0x00007ffff7a1ad6f in rustc_driver::driver::compile_input::he84fc71bb26c3ae0 ()
   from /home/nikic/rust/build/x86_64-unknown-linux-gnu/stage1/bin/../lib/librustc_driver-34cfa81bde6fac8c.so
#3  0x00007fffffff8600 in ?? ()
#4  0x00007fffffff8b30 in ?? ()
#5  0x0000000000000001 in ?? ()
#6  0x00007fffffff8501 in ?? ()
#7  0x00007ffff7ab5dd5 in rustc_driver::run_compiler_with_pool::hf7847454e1489610 ()
   from /home/nikic/rust/build/x86_64-unknown-linux-gnu/stage1/bin/../lib/librustc_driver-34cfa81bde6fac8c.so
#8  0x00007ffff79d8643 in syntax::with_globals::h3467ea0b3a675539 ()
   from /home/nikic/rust/build/x86_64-unknown-linux-gnu/stage1/bin/../lib/librustc_driver-34cfa81bde6fac8c.so
#9  0x00007ffff79b0a5e in _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::hbeda43bf54c2692e
    ()
   from /home/nikic/rust/build/x86_64-unknown-linux-gnu/stage1/bin/../lib/librustc_driver-34cfa81bde6fac8c.so
#10 0x00007ffff76da08a in __rust_maybe_catch_panic ()
   from /home/nikic/rust/build/x86_64-unknown-linux-gnu/stage1/bin/../lib/libstd-fc6b9a3d7065b2e2.so
#11 0x00007ffff7ab3064 in rustc_driver::run::h69cfea9e720cab41 ()
   from /home/nikic/rust/build/x86_64-unknown-linux-gnu/stage1/bin/../lib/librustc_driver-34cfa81bde6fac8c.so
#12 0x00007ffff7ac08fb in rustc_driver::main::h8abbba663fdf77b9 ()
   from /home/nikic/rust/build/x86_64-unknown-linux-gnu/stage1/bin/../lib/librustc_driver-34cfa81bde6fac8c.so
#13 0x0000555555554b23 in std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h6c799bafc2cc7997 ()
#14 0x00007ffff76a46b3 in std::panicking::try::do_call::h5d13bd17c519af69 ()
   from /home/nikic/rust/build/x86_64-unknown-linux-gnu/stage1/bin/../lib/libstd-fc6b9a3d7065b2e2.so
#15 0x00007ffff76da08a in __rust_maybe_catch_panic ()
   from /home/nikic/rust/build/x86_64-unknown-linux-gnu/stage1/bin/../lib/libstd-fc6b9a3d7065b2e2.so
#16 0x00007ffff7690486 in std::rt::lang_start_internal::he15214c0a1fdba94 ()
   from /home/nikic/rust/build/x86_64-unknown-linux-gnu/stage1/bin/../lib/libstd-fc6b9a3d7065b2e2.so
#17 0x0000555555554b84 in main ()

Sample panic while panicking:

stack backtrace:
   0:     0x7f8e3cd9bc2e - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::ha82b397ea71be585
   1:     0x7f8e3cd71796 - std::sys_common::backtrace::print::h0f02aefa57d1295d
   2:     0x7f8e3cd782bd - std::panicking::default_hook::{{closure}}::hb248860985868cdd
   3:     0x7f8e3cd78030 - std::panicking::default_hook::h0dd63170d2171fd8
   4:     0x7f8e393070d5 - rustc::util::common::panic_hook::h5a856d10e946cf82
   5:     0x7f8e3cd78943 - std::panicking::rust_panic_with_hook::h470a7ba29e7f6e22
   6:     0x7f8e3cd7853c - std::panicking::begin_panic_fmt::hdda42c41c88b4167
   7:     0x7f8e3cd78501 - rust_begin_unwind
   8:     0x7f8e3cdcf722 - core::panicking::panic_fmt::hd47ce5be3b71cd97
   9:     0x7f8e3d15c4a5 - core::result::unwrap_failed::h38f2e7889fea9e60
  10:     0x7f8e3d182df4 - <arena::TypedArena<T> as core::ops::drop::Drop>::drop::h694e59aa0a4a1d47
  11:     0x7f8e3d0da9cc - core::ptr::drop_in_place::h0d7fde6281af2c84
  12:     0x7f8e3d0eeaf9 - rustc_driver::driver::compile_input::he84fc71bb26c3ae0
  13:                0x3 - <unknown>
thread panicked while panicking. aborting.

What stands out in both cases is that the trace contains a low-value address.

[kennytm]

Just to confirm, is the libbacktrace submodule up to date? (git submodule update --init --recursive)

Is it still reproducible after ./x.py clean?

[nikic]

@kennytm Yeah, the submodule is up to date and this still reproduced after doing a ./x.py clean --all.

[kennytm]

Could you verify if rolling back to an earlier version of libbacktrace would fix the problem?

1. Make sure config.toml has [build] submodules = false to stop x.py from automatically managing submodules
2. cd src/libbacktrace
3. git checkout 14d377e9be7c89511c472d728e1b88b4e96f1946
4. cd ../..
5. Build again

[Mark-Simulacrum]

None of the UI tests fail for me on Ubuntu 18.04 with latest master. Can you also check that running git reset --hard inside src/libbacktrace doesn't fix the problem for you?

[nikic]

@kennytm Nope, did not help. Still seeing the same failures.

@Mark-Simulacrum I just did an rm -rf rust and checked out and rebuilt everything from scratch -- still same result :/

[Mark-Simulacrum]

Can you reproduce running ./src/ci/docker/run.sh x86_64-gnu? That should isolate the environment as much as possible...

[nikic]

@Mark-Simulacrum Docker ran through cleanly (only some network related test failures).

[Mark-Simulacrum]

Unfortunate. I'm not sure what could be the cause of this then...

[nikic]

I've switched to a fresh Ubuntu 18.04 VM now, where the problem doesn't occur. As it seems like I'm the only one seeing this, I guess we can just chalk this up to some kind of one-time environment corruption.

[vorner]

I'm seeing this on my gentoo system (162 failures in ui tests, as described above). Yesterday master (01cc982), fresh clone, no modifications to config.toml except for low-priotity = true.

I'd like to provide more info if I knew which one.

[nikic]

Btw, as another data point, before I gave up on this I also tried just reverting the build-system changes but keeping the updated libbacktrace. Result was the same.

[Mark-Simulacrum]

@vorner What version of Gentoo?

[vorner]

Gentoo is rolling-updates distribution, so it doesn't have versions as such. There's something called „profile“, which is a basic set of packages, but these packages still update one by one separately (my profile is default/linux/amd64/17.0). I keep relatively to the bleeding edge ‒ I did update the system during the last 2 weeks, so all the packages should be around what is current in the gentoo repositories.

But this might be of some use (this is part of what gentoo devs ask in bugreports):

Portage 2.3.40 (python 3.5.5-final-0, default/linux/amd64/17.0, gcc-6.4.0, glibc-2.27-r4, 4.15.2-hydra x86_64)
=================================================================
System uname: Linux-4.15.2-hydra-x86_64-AMD_FX-8370_Eight-Core_Processor-with-gentoo-2.6
KiB Mem:    32860244 total,   6051636 free
KiB Swap:   33552380 total,  33550588 free
Timestamp of repository gentoo: Sun, 24 Jun 2018 00:45:01 +0000
sh bash 4.4_p23
ld GNU ld (Gentoo 2.30 p3) 2.30.0
distcc 3.2rc1 x86_64-pc-linux-gnu [disabled]
ccache version 3.3.4 [disabled]
app-shells/bash:          4.4_p23::gentoo
dev-java/java-config:     2.2.0-r4::gentoo
dev-lang/perl:            5.26.2::gentoo
dev-lang/python:          2.7.14-r2::gentoo, 3.5.5-r1::gentoo, 3.6.5-r1::gentoo
dev-util/ccache:          3.3.4-r1::gentoo
dev-util/cmake:           3.11.2::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.6::gentoo
sys-apps/openrc:          0.37::gentoo
sys-apps/sandbox:         2.13::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69-r4::gentoo
sys-devel/automake:       1.11.6-r3::gentoo, 1.16.1-r1::gentoo
sys-devel/binutils:       2.30-r3::gentoo
sys-devel/gcc:            6.4.0-r1::gentoo, 7.3.0-r3::gentoo
sys-devel/gcc-config:     1.9.1::gentoo
sys-devel/libtool:        2.4.6-r5::gentoo
sys-devel/make:           4.2.1-r3::gentoo
sys-kernel/linux-headers: 4.17::gentoo (virtual/os-headers)
sys-libs/glibc:           2.27-r4::gentoo

From the installed gccs, the 6.4 is the default one.

[jyn514]

@nikic @vorner are you still seeing this issue?

[vorner]

This is indeed an old one, I no longer remember exact details.

But following my own info, I didn't reproduce the problem, all tests passed now. Seems to be fixed.

[jyn514]

🎉 glad to hear it :)