Long delay before exiting after reporting compilation errors in rustc 1.14.0 nightly when RUST_BACKTRACE=1 #37477
Labels
regression-from-stable-to-stable
Performance or correctness regression from one stable version to another.
T-libs-api
Relevant to the library API team, which will review and decide on the PR/issue.
Comments
|
If it helps, that same version of rustc on Windows 10 works OK for me, there is no noticeable delay after compilation error reporting. |
|
Actually I failed to report that I was using Ubuntu 16.04 virtual machine on VMware Workstation 12 (most recent version) on native Windows 10. I just have tested for this issue on Ubuntu 16.04 on bare metal, and rustc worked fine. The issue seems to be related to virtualization somehow. |
jchlapinski
changed the title
|
Note that the compiler continues after it hits the first error, so the compiler could possibly have been doing real compilation work (looking for more errors), so perhaps that's what's happening here? |
|
Have you considered profiling rustc to see where it is spending most of those 13 seconds? |
|
@retep998 here is what I've gathered from 'perf' the results from my Ubuntu 16.04 virtual machine running on VMware Workstation (only few lines from the top for brevity): Clearly the problem is I will also put here the results from bare metal for comparison. |
|
results for Ubuntu 16.04 running bare metal: |
|
I have just realized that on my virtual machine I had environment variable RUST_BACKTRACE=1, whereas on my bare-metal I had not. Indeed when setting RUST_BACKTRACE=1 on both machines the results are more or less the same (there is a very long delay before rustc exits). So the problem is not with virtualization after all. Definitely the problem lays in backtrace code somewhere. I started to notice this issue somewhere around a week ago, as I am updating rust nightly very often (daily). |
jchlapinski
changed the title
|
Possibly duplicate of #29293 |
This was referenced Nov 19, 2016
|
1.14 beta exhibits this issue. If it goes stable it will be a significant headache/puzzle for everyone who leaves RUST_BACKTRACE set. cc @nikomatsakis #37571 seems to be a duplicate issue. In the same 10/25 nightly that introduced this, the release hello world executable size on |
|
This is negatively impacting the alternate playground, which always sets RUST_BACKTRACE, and is now present in released 1.14. |
shepmaster
added a commit
to rust-lang/rust-playground
that referenced
this issue
Dec 25, 2016
|
|
Hey, this has made it's way to stable and it is a rather serious problem I think. @alexcrichton, perhaps it should have some labels/assignee applied? We face it in IntelliJ Rust: intellij-rust/intellij-rust#867 We run cargo commands with If it is useful, here's a flamegraph of the hard working rustc: http://svgshare.com/i/TU.svg |
|
@matklad sounds reasonable to me. I'll nominate for discussion in libs triage as I believe this is basically entirely about libbacktrace |
alexcrichton
added
I-nominated
T-libs-api
|
What makes libbacktrace so slow, anyway? I've definitely noticed the very long lags in the past. |
Judging by the pretty flamegraph and the perf report, I'm going to guess that something is being eagerly allocated when it doesn't need to be. |
|
I've tried to track this down, but been unable to reproduce with nightly on macOS 10.12.2 or stage1 and stage2 compilers on Ubuntu 16.04 (on 95b14a3). While I do notice a slight increase (~100-400 ms) when either the compiler ICEs or a crate panics, I sort of expect that since more work is being done. However, I can reproduce with nightly (rustc 1.16.0-nightly (4682271 2017-01-03)) on Ubuntu 16.04, which sort of confuses me. I'm not sure what the difference between those and stage1/stage2 builds on the same machine is. To summarize:
This makes benchmarking and testing changes difficult/impossible, since I can't reproduce with a locally built compiler. I'm not really sure what can be said here. I also confirmed that this is not due to the wrapper script that rustbuild introduces. Exact measurements below. macOS 10.12.2, rustc 1.16.0-nightly (4682271 2017-01-03). Ubuntu 16.04, rustc 1.16.0-nightly (4682271 2017-01-03) -- same as on macOS Ubuntu 16.04, Stage 1 Compiler ( Ubuntu 16.04, Stage 2 Compiler ( It may also be of interested that utilizing strace, it seems that the stage1/stage2 compilers benchmarked above issue hundreds of mmap calls, approximately page-by-page ( Hopefully all of this can help narrow this issue down. I'd be happy to collect any other statistics that could be helpful. |
|
@Mark-Simulacrum interesting! I wonder if this means it's all related to how we're building the nightlies. The main differences I can think of there are:
Now that I list these out, did you enable debuginfo in the compiler you produced locally? IIRC that was a historical problem and I don't think we ever fixed that. |
I see delay even if |
This occurs with Rust 1.14 stable on Linux for me. |
|
@shepmaster oh I should say releases*, not just nightlies. They're all built the same way modulo llvm-assertions (which we can now rule out if it's a problem on stable) |
|
FWIW, here's the reproduction steps I have: |
|
For some reason, this reminds me of rust-lang/rustup#783. Most likely unrelated, but symptoms are somewhat similar. |
bors
added a commit
that referenced
this issue
Jan 11, 2017
rustbuild: Don't enable debuginfo in rustc In #37280 we enabled line number debugging information in release artifacts, primarily to close out #36452 where debugging information was critical for MSVC builds of Rust to be useful in production. This commit, however, apparently had some unfortunate side effects. Namely it was noticed in #37477 that if `RUST_BACKTRACE=1` was set then any compiler error would take a very long time for the compiler to exit. The cause of the problem here was somewhat deep: * For all compiler errors, the compiler will `panic!` with a known value. This tears down the main compiler thread and allows cleaning up all the various resources. By default, however, this panic output is suppressed for "normal" compiler errors. * When `RUST_BACKTRACE=1` was set this caused every compiler error to generate a backtrace. * The libbacktrace library hits a pathological case where it spends a very long time in its custom allocation function, `backtrace_alloc`, because the compiler has so much debugging information. More information about this can be found in #29293 with a summary at the end of #37477. To solve this problem this commit simply removes debuginfo from the compiler but not from the standard library. This should allow us to keep #36452 closed while also closing #37477. I've measured the difference to be orders of magnitude faster than it was before, so we should see a much quicker time-to-exit after a compile error when `RUST_BACKTRACE=1` is set. Closes #37477 Closes #37571
|
@Mark-Simulacrum oh sorry #38984 landed more quickly than I thought it would! To clarify I personally prefer to not change libbacktrace much to make future vendoring efforts easier, but that's mostly just me. |
|
What I really want is for us to adopt some kind of library that works across platforms, so that mac can enjoy line numbers too! (I'm not sure of the status on windows here...) |
brson
pushed a commit
to brson/rust
that referenced
this issue
Jan 17, 2017
In rust-lang#37280 we enabled line number debugging information in release artifacts, primarily to close out rust-lang#36452 where debugging information was critical for MSVC builds of Rust to be useful in production. This commit, however, apparently had some unfortunate side effects. Namely it was noticed in rust-lang#37477 that if `RUST_BACKTRACE=1` was set then any compiler error would take a very long time for the compiler to exit. The cause of the problem here was somewhat deep: * For all compiler errors, the compiler will `panic!` with a known value. This tears down the main compiler thread and allows cleaning up all the various resources. By default, however, this panic output is suppressed for "normal" compiler errors. * When `RUST_BACKTRACE=1` was set this caused every compiler error to generate a backtrace. * The libbacktrace library hits a pathological case where it spends a very long time in its custom allocation function, `backtrace_alloc`, because the compiler has so much debugging information. More information about this can be found in rust-lang#29293 with a summary at the end of rust-lang#37477. To solve this problem this commit simply removes debuginfo from the compiler but not from the standard library. This should allow us to keep rust-lang#36452 closed while also closing rust-lang#37477. I've measured the difference to be orders of magnitude faster than it was before, so we should see a much quicker time-to-exit after a compile error when `RUST_BACKTRACE=1` is set. Closes rust-lang#37477 Closes rust-lang#37571
ghost
mentioned this issue
alexcrichton
added a commit
to alexcrichton/rust
that referenced
this issue
Oct 25, 2017
This is sort of a long overdue change from the investigation in rust-lang#29293 and rust-lang#37477. The released binaries of rustc don't have debug information and so don't actively suffer this problem but this can hit local development of rustc and also larger programs compiled against libstd generating backtraces. The main purpose of the mmap allocator in libacktrace is to be usable from a signal handler, but we don't do that, so the normal allocator using malloc/free should work well for us.
bors
added a commit
that referenced
this issue
Oct 26, 2017
std: Disable usage of mmap allocator in libbacktrace This is sort of a long overdue change from the investigation in #29293 and #37477. The released binaries of rustc don't have debug information and so don't actively suffer this problem but this can hit local development of rustc and also larger programs compiled against libstd generating backtraces. The main purpose of the mmap allocator in libacktrace is to be usable from a signal handler, but we don't do that, so the normal allocator using malloc/free should work well for us.
ianlancetaylor
added a commit
to ianlancetaylor/libbacktrace
that referenced
this issue
Jan 25, 2018
on the free list. Fixes #5 Fixes rust-lang/rust#29293 Fixes rust-lang/rust#37477
Merged
6 tasks
alexcrichton
added a commit
to alexcrichton/rust
that referenced
this issue
Jul 17, 2020
This commit is a proof-of-concept for switching the standard library's backtrace symbolication mechanism on most platforms from libbacktrace to gimli. The standard library's support for `RUST_BACKTRACE=1` requires in-process parsing of object files and DWARF debug information to interpret it and print the filename/line number of stack frames as part of a backtrace. Historically this support in the standard library has come from a library called "libbacktrace". The libbacktrace library seems to have been extracted from gcc at some point and is written in C. We've had a lot of issues with libbacktrace over time, unfortunately, though. The library does not appear to be actively maintained since we've had patches sit for months-to-years without comments. We have discovered a good number of soundness issues with the library itself, both when parsing valid DWARF as well as invalid DWARF. This is enough of an issue that the libs team has previously decided that we cannot feed untrusted inputs to libbacktrace. This also doesn't take into account the portability of libbacktrace which has been difficult to manage and maintain over time. While possible there are lots of exceptions and it's the main C dependency of the standard library right now. For years it's been the desire to switch over to a Rust-based solution for symbolicating backtraces. It's been assumed that we'll be using the Gimli family of crates for this purpose, which are targeted at safely and efficiently parsing DWARF debug information. I've been working recently to shore up the Gimli support in the `backtrace` crate. As of a few weeks ago the `backtrace` crate, by default, uses Gimli when loaded from crates.io. This transition has gone well enough that I figured it was time to start talking seriously about this change to the standard library. This commit is a preview of what's probably the best way to integrate the `backtrace` crate into the standard library with the Gimli feature turned on. While today it's used as a crates.io dependency, this commit switches the `backtrace` crate to a submodule of this repository which will need to be updated manually. This is not done lightly, but is thought to be the best solution. The primary reason for this is that the `backtrace` crate needs to do some pretty nontrivial filesystem interactions to locate debug information. Working without `std::fs` is not an option, and while it might be possible to do some sort of trait-based solution when prototyped it was found to be too unergonomic. Using a submodule allows the `backtrace` crate to build as a submodule of the `std` crate itself, enabling it to use `std::fs` and such. Otherwise this adds new dependencies to the standard library. This step requires extra attention because this means that these crates are now going to be included with all Rust programs by default. It's important to note, however, that we're already shipping libbacktrace with all Rust programs by default and it has a bunch of C code implementing all of this internally anyway, so we're basically already switching already-shipping functionality to Rust from C. * `object` - this crate is used to parse object file headers and contents. Very low-level support is used from this crate and almost all of it is disabled. Largely we're just using struct definitions as well as convenience methods internally to read bytes and such. * `addr2line` - this is the main meat of the implementation for symbolication. This crate depends on `gimli` for DWARF parsing and then provides interfaces needed by the `backtrace` crate to turn an address into a filename / line number. This crate is actually pretty small (fits in a single file almost!) and mirrors most of what `dwarf.c` does for libbacktrace. * `miniz_oxide` - the libbacktrace crate transparently handles compressed debug information which is compressed with zlib. This crate is used to decompress compressed debug sections. * `gimli` - not actually used directly, but a dependency of `addr2line`. * `adler32`- not used directly either, but a dependency of `miniz_oxide`. The goal of this change is to improve the safety of backtrace symbolication in the standard library, especially in the face of possibly malformed DWARF debug information. Even to this day we're still seeing segfaults in libbacktrace which could possibly become security vulnerabilities. This change should almost entirely eliminate this possibility whilc also paving the way forward to adding more features like split debug information. Some references for those interested are: * Original addition of libbacktrace - rust-lang#12602 * OOM with libbacktrace - rust-lang#24231 * Backtrace failure due to use of uninitialized value - rust-lang#28447 * Possibility to feed untrusted data to libbacktrace - rust-lang#21889 * Soundness fix for libbacktrace - rust-lang#33729 * Crash in libbacktrace - rust-lang#39468 * Support for macOS, never merged - ianlancetaylor/libbacktrace#2 * Performance issues with libbacktrace - rust-lang#29293, rust-lang#37477 * Update procedure is quite complicated due to how many patches we need to carry - rust-lang#50955 * Libbacktrace doesn't work on MinGW with dynamic libs - rust-lang#71060 * Segfault in libbacktrace on macOS - rust-lang#71397 Switching to Rust will not make us immune to all of these issues. The crashes are expected to go away, but correctness and performance may still have bugs arise. The gimli and `backtrace` crates, however, are actively maintained unlike libbacktrace, so this should enable us to at least efficiently apply fixes as situations come up.
bors
added a commit
to rust-lang-ci/rust
that referenced
this issue
Jul 18, 2020
…Simulacrum std: Switch from libbacktrace to gimli This commit is a proof-of-concept for switching the standard library's backtrace symbolication mechanism on most platforms from libbacktrace to gimli. The standard library's support for `RUST_BACKTRACE=1` requires in-process parsing of object files and DWARF debug information to interpret it and print the filename/line number of stack frames as part of a backtrace. Historically this support in the standard library has come from a library called "libbacktrace". The libbacktrace library seems to have been extracted from gcc at some point and is written in C. We've had a lot of issues with libbacktrace over time, unfortunately, though. The library does not appear to be actively maintained since we've had patches sit for months-to-years without comments. We have discovered a good number of soundness issues with the library itself, both when parsing valid DWARF as well as invalid DWARF. This is enough of an issue that the libs team has previously decided that we cannot feed untrusted inputs to libbacktrace. This also doesn't take into account the portability of libbacktrace which has been difficult to manage and maintain over time. While possible there are lots of exceptions and it's the main C dependency of the standard library right now. For years it's been the desire to switch over to a Rust-based solution for symbolicating backtraces. It's been assumed that we'll be using the Gimli family of crates for this purpose, which are targeted at safely and efficiently parsing DWARF debug information. I've been working recently to shore up the Gimli support in the `backtrace` crate. As of a few weeks ago the `backtrace` crate, by default, uses Gimli when loaded from crates.io. This transition has gone well enough that I figured it was time to start talking seriously about this change to the standard library. This commit is a preview of what's probably the best way to integrate the `backtrace` crate into the standard library with the Gimli feature turned on. While today it's used as a crates.io dependency, this commit switches the `backtrace` crate to a submodule of this repository which will need to be updated manually. This is not done lightly, but is thought to be the best solution. The primary reason for this is that the `backtrace` crate needs to do some pretty nontrivial filesystem interactions to locate debug information. Working without `std::fs` is not an option, and while it might be possible to do some sort of trait-based solution when prototyped it was found to be too unergonomic. Using a submodule allows the `backtrace` crate to build as a submodule of the `std` crate itself, enabling it to use `std::fs` and such. Otherwise this adds new dependencies to the standard library. This step requires extra attention because this means that these crates are now going to be included with all Rust programs by default. It's important to note, however, that we're already shipping libbacktrace with all Rust programs by default and it has a bunch of C code implementing all of this internally anyway, so we're basically already switching already-shipping functionality to Rust from C. * `object` - this crate is used to parse object file headers and contents. Very low-level support is used from this crate and almost all of it is disabled. Largely we're just using struct definitions as well as convenience methods internally to read bytes and such. * `addr2line` - this is the main meat of the implementation for symbolication. This crate depends on `gimli` for DWARF parsing and then provides interfaces needed by the `backtrace` crate to turn an address into a filename / line number. This crate is actually pretty small (fits in a single file almost!) and mirrors most of what `dwarf.c` does for libbacktrace. * `miniz_oxide` - the libbacktrace crate transparently handles compressed debug information which is compressed with zlib. This crate is used to decompress compressed debug sections. * `gimli` - not actually used directly, but a dependency of `addr2line`. * `adler32`- not used directly either, but a dependency of `miniz_oxide`. The goal of this change is to improve the safety of backtrace symbolication in the standard library, especially in the face of possibly malformed DWARF debug information. Even to this day we're still seeing segfaults in libbacktrace which could possibly become security vulnerabilities. This change should almost entirely eliminate this possibility whilc also paving the way forward to adding more features like split debug information. Some references for those interested are: * Original addition of libbacktrace - rust-lang#12602 * OOM with libbacktrace - rust-lang#24231 * Backtrace failure due to use of uninitialized value - rust-lang#28447 * Possibility to feed untrusted data to libbacktrace - rust-lang#21889 * Soundness fix for libbacktrace - rust-lang#33729 * Crash in libbacktrace - rust-lang#39468 * Support for macOS, never merged - ianlancetaylor/libbacktrace#2 * Performance issues with libbacktrace - rust-lang#29293, rust-lang#37477 * Update procedure is quite complicated due to how many patches we need to carry - rust-lang#50955 * Libbacktrace doesn't work on MinGW with dynamic libs - rust-lang#71060 * Segfault in libbacktrace on macOS - rust-lang#71397 Switching to Rust will not make us immune to all of these issues. The crashes are expected to go away, but correctness and performance may still have bugs arise. The gimli and `backtrace` crates, however, are actively maintained unlike libbacktrace, so this should enable us to at least efficiently apply fixes as situations come up. --- I want to note that my purpose for creating a PR here is to start a conversation about this. I think that all the various pieces are in place that this is compelling enough that I think this transition should be talked about seriously. There are a number of items which still need to be addressed before actually merging this PR, however: * [ ] `gimli` needs to be published to crates.io * [ ] `addr2line` needs a publish * [ ] `miniz_oxide` needs a publish * [ ] Tests probably shouldn't recommend the `gimli` crate's traits for implementing * [ ] The `backtrace` crate's branch changes need to be merged to the master branch (rust-lang/backtrace-rs#349) * [ ] The support for `libbacktrace` on some platforms needs to be audited to see if we should support more strategies in the gimli implementation - rust-lang/backtrace-rs#325, rust-lang/backtrace-rs#326, rust-lang/backtrace-rs#350, rust-lang/backtrace-rs#351 Most of the merging/publishing I'm not actively pushing on right now. It's a bit wonky for crates to support libstd so I'm holding off on pulling the trigger everywhere until there's a bit more discussion about how to go through with this. Namely rust-lang/backtrace-rs#349 I'm going to hold off merging until we decide to go through with the submodule strategy. In any case this is a pretty major change, so I suspect that the compiler team is likely going to be interested in this. I don't mean to force changes by dumping a bunch of code by any means. Integration of external crates into the standard library is so difficult I wanted to have a proof-of-concept to review while talking about whether to do this at all (hence the PR), but I'm more than happy to follow any processes needed to merge this. I must admit though that I'm not entirely sure myself at this time what the process would be to decide to merge this, so I'm hoping others can help me figure that out!
alexcrichton
added a commit
to alexcrichton/rust
that referenced
this issue
Jul 28, 2020
This commit is a proof-of-concept for switching the standard library's backtrace symbolication mechanism on most platforms from libbacktrace to gimli. The standard library's support for `RUST_BACKTRACE=1` requires in-process parsing of object files and DWARF debug information to interpret it and print the filename/line number of stack frames as part of a backtrace. Historically this support in the standard library has come from a library called "libbacktrace". The libbacktrace library seems to have been extracted from gcc at some point and is written in C. We've had a lot of issues with libbacktrace over time, unfortunately, though. The library does not appear to be actively maintained since we've had patches sit for months-to-years without comments. We have discovered a good number of soundness issues with the library itself, both when parsing valid DWARF as well as invalid DWARF. This is enough of an issue that the libs team has previously decided that we cannot feed untrusted inputs to libbacktrace. This also doesn't take into account the portability of libbacktrace which has been difficult to manage and maintain over time. While possible there are lots of exceptions and it's the main C dependency of the standard library right now. For years it's been the desire to switch over to a Rust-based solution for symbolicating backtraces. It's been assumed that we'll be using the Gimli family of crates for this purpose, which are targeted at safely and efficiently parsing DWARF debug information. I've been working recently to shore up the Gimli support in the `backtrace` crate. As of a few weeks ago the `backtrace` crate, by default, uses Gimli when loaded from crates.io. This transition has gone well enough that I figured it was time to start talking seriously about this change to the standard library. This commit is a preview of what's probably the best way to integrate the `backtrace` crate into the standard library with the Gimli feature turned on. While today it's used as a crates.io dependency, this commit switches the `backtrace` crate to a submodule of this repository which will need to be updated manually. This is not done lightly, but is thought to be the best solution. The primary reason for this is that the `backtrace` crate needs to do some pretty nontrivial filesystem interactions to locate debug information. Working without `std::fs` is not an option, and while it might be possible to do some sort of trait-based solution when prototyped it was found to be too unergonomic. Using a submodule allows the `backtrace` crate to build as a submodule of the `std` crate itself, enabling it to use `std::fs` and such. Otherwise this adds new dependencies to the standard library. This step requires extra attention because this means that these crates are now going to be included with all Rust programs by default. It's important to note, however, that we're already shipping libbacktrace with all Rust programs by default and it has a bunch of C code implementing all of this internally anyway, so we're basically already switching already-shipping functionality to Rust from C. * `object` - this crate is used to parse object file headers and contents. Very low-level support is used from this crate and almost all of it is disabled. Largely we're just using struct definitions as well as convenience methods internally to read bytes and such. * `addr2line` - this is the main meat of the implementation for symbolication. This crate depends on `gimli` for DWARF parsing and then provides interfaces needed by the `backtrace` crate to turn an address into a filename / line number. This crate is actually pretty small (fits in a single file almost!) and mirrors most of what `dwarf.c` does for libbacktrace. * `miniz_oxide` - the libbacktrace crate transparently handles compressed debug information which is compressed with zlib. This crate is used to decompress compressed debug sections. * `gimli` - not actually used directly, but a dependency of `addr2line`. * `adler32`- not used directly either, but a dependency of `miniz_oxide`. The goal of this change is to improve the safety of backtrace symbolication in the standard library, especially in the face of possibly malformed DWARF debug information. Even to this day we're still seeing segfaults in libbacktrace which could possibly become security vulnerabilities. This change should almost entirely eliminate this possibility whilc also paving the way forward to adding more features like split debug information. Some references for those interested are: * Original addition of libbacktrace - rust-lang#12602 * OOM with libbacktrace - rust-lang#24231 * Backtrace failure due to use of uninitialized value - rust-lang#28447 * Possibility to feed untrusted data to libbacktrace - rust-lang#21889 * Soundness fix for libbacktrace - rust-lang#33729 * Crash in libbacktrace - rust-lang#39468 * Support for macOS, never merged - ianlancetaylor/libbacktrace#2 * Performance issues with libbacktrace - rust-lang#29293, rust-lang#37477 * Update procedure is quite complicated due to how many patches we need to carry - rust-lang#50955 * Libbacktrace doesn't work on MinGW with dynamic libs - rust-lang#71060 * Segfault in libbacktrace on macOS - rust-lang#71397 Switching to Rust will not make us immune to all of these issues. The crashes are expected to go away, but correctness and performance may still have bugs arise. The gimli and `backtrace` crates, however, are actively maintained unlike libbacktrace, so this should enable us to at least efficiently apply fixes as situations come up.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When I have any compilation error(s) there is a rather long delay (around 14s on my machine) before rustc finally exits. Successful compilation is very fast.
I am on Ubuntu 16.04 in VMware Workstation 12.5 for Windows virtual machine.
Steps to reproduce:
Meta
rustc --version --verbose:The text was updated successfully, but these errors were encountered: