Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

thread 'rustc' panicked at /rustc/1388d7a069d872bcfe5e5dd97ef61fa0a586fac0/compiler/rustc_query_system/src/dep_graph/serialized.rs:190:9: #123352

Closed
lengyijun opened this issue Apr 2, 2024 · 8 comments
Assignees
Labels
A-incr-comp Area: Incremental compilation C-bug Category: This is a bug. I-ICE Issue: The compiler panicked, giving an Internal Compilation Error (ICE) ❄️ T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@lengyijun
Copy link
Contributor

Code

https://github.com/lengyijun/rust-clippy/tree/pathbuf_join
510bcef579cad260cfac0aa305262550ced71caa

Meta

rustc --version --verbose:

❯ rustc --version --verbose
rustc 1.79.0-nightly (1388d7a06 2024-03-20)
binary: rustc
commit-hash: 1388d7a069d872bcfe5e5dd97ef61fa0a586fac0
commit-date: 2024-03-20
host: x86_64-unknown-linux-gnu
release: 1.79.0-nightly
LLVM version: 18.1.2

Error output

<output>
Backtrace

thread 'rustc' panicked at /rustc/1388d7a069d872bcfe5e5dd97ef61fa0a586fac0/compiler/rustc_query_system/src/dep_graph/serialized.rs:190:9:
    assertion `left == right` failed
      left: 7421952
     right: 111465868407341450
    stack backtrace:
       0:     0x7f5551d8bfa5 - std::backtrace_rs::backtrace::libunwind::trace::h532ed8ab333aacce
                                   at /rustc/1388d7a069d872bcfe5e5dd97ef61fa0a586fac0/library/std/src/../../backtrace/src/backtrace/libunwind.rs:105:5
       1:     0x7f5551d8bfa5 - std::backtrace_rs::backtrace::trace_unsynchronized::ha49c9c586fc8ba0d
                                   at /rustc/1388d7a069d872bcfe5e5dd97ef61fa0a586fac0/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
       2:     0x7f5551d8bfa5 - std::sys_common::backtrace::_print_fmt::hda98b14c7195d752
                                   at /rustc/1388d7a069d872bcfe5e5dd97ef61fa0a586fac0/library/std/src/sys_common/backtrace.rs:68:5
       3:     0x7f5551d8bfa5 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h4e3932cf0d29e118
                                   at /rustc/1388d7a069d872bcfe5e5dd97ef61fa0a586fac0/library/std/src/sys_common/backtrace.rs:44:22
       4:     0x7f5551ddcfeb - core::fmt::rt::Argument::fmt::hd76e6e38a085e245
                                   at /rustc/1388d7a069d872bcfe5e5dd97ef61fa0a586fac0/library/core/src/fmt/rt.rs:142:9
       5:     0x7f5551ddcfeb - core::fmt::write::hdf2c6134be67eb54
                                   at /rustc/1388d7a069d872bcfe5e5dd97ef61fa0a586fac0/library/core/src/fmt/mod.rs:1153:17
       6:     0x7f5551d80cbf - std::io::Write::write_fmt::h31850499d879e043
                                   at /rustc/1388d7a069d872bcfe5e5dd97ef61fa0a586fac0/library/std/src/io/mod.rs:1843:15
       7:     0x7f5551d8bd7e - std::sys_common::backtrace::_print::he7f8a7af0244fed2
                                   at /rustc/1388d7a069d872bcfe5e5dd97ef61fa0a586fac0/library/std/src/sys_common/backtrace.rs:47:5
       8:     0x7f5551d8bd7e - std::sys_common::backtrace::print::h1ada40475248e895
                                   at /rustc/1388d7a069d872bcfe5e5dd97ef61fa0a586fac0/library/std/src/sys_common/backtrace.rs:34:9
       9:     0x7f5551d8e9f9 - std::panicking::default_hook::{{closure}}::h229d3e1bad8bb4bc
      10:     0x7f5551d8e763 - std::panicking::default_hook::hf22d5baf4d45616f
                                   at /rustc/1388d7a069d872bcfe5e5dd97ef61fa0a586fac0/library/std/src/panicking.rs:292:9
      11:     0x7f5554dbe95e - std[ef9c3a509fa4eaac]::panicking::update_hook::<alloc[c1b873ed893cf9ea]::boxed::Box<rustc_driver_impl[25d18c121fdceb6a]::install_ice_hook::{closure#0}>>::{closure#0}
      12:     0x7f5551d8f150 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h34e6b4199b741f54
                                   at /rustc/1388d7a069d872bcfe5e5dd97ef61fa0a586fac0/library/alloc/src/boxed.rs:2029:9
      13:     0x7f5551d8f150 - std::panicking::rust_panic_with_hook::h56dd3ea19c089233
                                   at /rustc/1388d7a069d872bcfe5e5dd97ef61fa0a586fac0/library/std/src/panicking.rs:783:13
      14:     0x7f5551d8eea6 - std::panicking::begin_panic_handler::{{closure}}::h1131cc0136fb1270
                                   at /rustc/1388d7a069d872bcfe5e5dd97ef61fa0a586fac0/library/std/src/panicking.rs:657:13
      15:     0x7f5551d8c469 - std::sys_common::backtrace::__rust_end_short_backtrace::h80f7d682167aaa2d
                                   at /rustc/1388d7a069d872bcfe5e5dd97ef61fa0a586fac0/library/std/src/sys_common/backtrace.rs:171:18
      16:     0x7f5551d8ebd7 - rust_begin_unwind
                                   at /rustc/1388d7a069d872bcfe5e5dd97ef61fa0a586fac0/library/std/src/panicking.rs:645:5
      17:     0x7f5551dd9486 - core::panicking::panic_fmt::h265b447c3aa2fb06
                                   at /rustc/1388d7a069d872bcfe5e5dd97ef61fa0a586fac0/library/core/src/panicking.rs:72:14
      18:     0x7f5551dd997e - core::panicking::assert_failed_inner::he5a3eea7c3e3a68c
                                   at /rustc/1388d7a069d872bcfe5e5dd97ef61fa0a586fac0/library/core/src/panicking.rs:339:17
      19:     0x7f5554ffb523 - core[154fb0d77e60979c]::panicking::assert_failed::<usize, usize>
      20:     0x7f5556d3efc0 - rustc_incremental[6b5ea6b7b98ff968]::persist::load::setup_dep_graph
      21:     0x7f5556da69bf - <rustc_interface[bd7448d5f7b292b]::queries::Queries>::global_ctxt
      22:     0x7f5556c91cd0 - rustc_interface[bd7448d5f7b292b]::interface::run_compiler::<core[154fb0d77e60979c]::result::Result<(), rustc_span[74e38aa60d99f81]::ErrorGuaranteed>, rustc_driver_impl[25d18c121fdceb6a]::run_compiler::{closure#0}>::{closure#0}
      23:     0x7f5556d8ef17 - std[ef9c3a509fa4eaac]::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface[bd7448d5f7b292b]::util::run_in_thread_with_globals<rustc_interface[bd7448d5f7b292b]::util::run_in_thread_pool_with_globals<rustc_interface[bd7448d5f7b292b]::interface::run_compiler<core[154fb0d77e60979c]::result::Result<(), rustc_span[74e38aa60d99f81]::ErrorGuaranteed>, rustc_driver_impl[25d18c121fdceb6a]::run_compiler::{closure#0}>::{closure#0}, core[154fb0d77e60979c]::result::Result<(), rustc_span[74e38aa60d99f81]::ErrorGuaranteed>>::{closure#0}, core[154fb0d77e60979c]::result::Result<(), rustc_span[74e38aa60d99f81]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[154fb0d77e60979c]::result::Result<(), rustc_span[74e38aa60d99f81]::ErrorGuaranteed>>
      24:     0x7f5556d8ed44 - <<std[ef9c3a509fa4eaac]::thread::Builder>::spawn_unchecked_<rustc_interface[bd7448d5f7b292b]::util::run_in_thread_with_globals<rustc_interface[bd7448d5f7b292b]::util::run_in_thread_pool_with_globals<rustc_interface[bd7448d5f7b292b]::interface::run_compiler<core[154fb0d77e60979c]::result::Result<(), rustc_span[74e38aa60d99f81]::ErrorGuaranteed>, rustc_driver_impl[25d18c121fdceb6a]::run_compiler::{closure#0}>::{closure#0}, core[154fb0d77e60979c]::result::Result<(), rustc_span[74e38aa60d99f81]::ErrorGuaranteed>>::{closure#0}, core[154fb0d77e60979c]::result::Result<(), rustc_span[74e38aa60d99f81]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[154fb0d77e60979c]::result::Result<(), rustc_span[74e38aa60d99f81]::ErrorGuaranteed>>::{closure#1} as core[154fb0d77e60979c]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
      25:     0x7f5551d98919 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h3b32cecf95e1137a
                                   at /rustc/1388d7a069d872bcfe5e5dd97ef61fa0a586fac0/library/alloc/src/boxed.rs:2015:9
      26:     0x7f5551d98919 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hf710b728c88ec181
                                   at /rustc/1388d7a069d872bcfe5e5dd97ef61fa0a586fac0/library/alloc/src/boxed.rs:2015:9
      27:     0x7f5551d98919 - std::sys::pal::unix::thread::Thread::new::thread_start::h678a890b8138c240
                                   at /rustc/1388d7a069d872bcfe5e5dd97ef61fa0a586fac0/library/std/src/sys/pal/unix/thread.rs:108:17
      28:     0x7f5551b4f55a - <unknown>
      29:     0x7f5551bcca3c - <unknown>
      30:                0x0 - <unknown>

    error: the compiler unexpectedly panicked. this is a bug.

    note: it seems that this compiler `1.79.0-nightly (1388d7a06 2024-03-20)` is outdated, a newer nightly should have been released in the mean time
      |
      = note: please consider running `rustup update nightly` to update the nightly channel and check if this problem still persists
      = note: if the problem still persists, we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

    note: please attach the file at `/home/lengyijun/rust-clippy-path/rustc-ice-2024-04-02T04_47_06-10586.txt` to your bug report

    note: compiler flags: --crate-type bin -C embed-bitcode=no -C debuginfo=2 -C split-debuginfo=unpacked -C incremental=[REDACTED] -Z unstable-options -Z binary-dep-depinfo -Z binary-dep-depinfo

    note: some of the compiler flags provided by cargo are hidden

    query stack during panic:
    end of query stack
    error: could not compile `clippy` (bin "clippy-driver")
    warning: build failed, waiting for other jobs to finish...

@lengyijun lengyijun added C-bug Category: This is a bug. I-ICE Issue: The compiler panicked, giving an Internal Compilation Error (ICE) ❄️ T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Apr 2, 2024
@rustbot rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Apr 2, 2024
@lengyijun
Copy link
Contributor Author

fixed after cargo clean

@jieyouxu jieyouxu removed the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Apr 2, 2024
@saethlin saethlin reopened this Apr 3, 2024
@saethlin saethlin added the A-incr-comp Area: Incremental compilation label Apr 3, 2024
@saethlin
Copy link
Member

saethlin commented Apr 3, 2024

It's great that you managed to get your own workflow going by running cargo clean, but that papers over the problem. Crashes like this are not acceptable compiler behavior. I added this assertion in order to more clearly detect a bug if there was one, and clearly there is.

@saethlin saethlin self-assigned this Apr 3, 2024
@lengyijun
Copy link
Contributor Author

lengyijun commented Apr 4, 2024

this panic happens after I fix an out-of-disk problem

@saethlin
Copy link
Member

saethlin commented Apr 4, 2024

Ooooo I thought I fixed all of the out-of-disk problems in 1.77. I'll take a look again.

@saethlin
Copy link
Member

saethlin commented Apr 8, 2024

this panic happens after I fix an out-of-disk problem

Ah I see you edited your message. What do you mean by "fix" here? Did you run out of disk during a build, then free up some space, then try to compile again and get this crash?

@lengyijun
Copy link
Contributor Author

lengyijun commented Apr 8, 2024

this panic happens after I fix an out-of-disk problem

Ah I see you edited your message. What do you mean by "fix" here? Did you run out of disk during a build, then free up some space, then try to compile again and get this crash?

IIRC, this is the case

I'm working on a proxmox virtual machine
I notice proxmox's io-error (a yellow symbol) when I was using rust-analyzer in vscode,
then cleanup disk, with a lot of reboots
then compile, then panic

@lukas-code
Copy link
Member

I've just hit the same assertion after recompiling after a power failure.

I think we may need to call fsync / File::sync_all on the dep-graph.part.bin file before renaming it to dep-graph.bin, otherwise data might get lost.

backtrace
thread 'rustc' panicked at /rustc/efd9d2df12b5e17fac0b4d0fb89f612ecd79f259/compiler/rustc_query_system/src/dep_graph/serialized.rs:190:9:
assertion `left == right` failed
  left: 30725197
 right: 32962287060911799
stack backtrace:
   0: rust_begin_unwind
             at /rustc/efd9d2df12b5e17fac0b4d0fb89f612ecd79f259/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/efd9d2df12b5e17fac0b4d0fb89f612ecd79f259/library/core/src/panicking.rs:72:14
   2: core::panicking::assert_failed_inner
             at /rustc/efd9d2df12b5e17fac0b4d0fb89f612ecd79f259/library/core/src/panicking.rs:343:17
   3: core::panicking::assert_failed::<usize, usize>
   4: rustc_incremental::persist::load::setup_dep_graph
   5: <rustc_interface::queries::Queries>::global_ctxt
   6: rustc_interface::interface::run_compiler::<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#0}>::{closure#0}

dep-graph.bin.zip

bors added a commit to rust-lang-ci/rust that referenced this issue Apr 13, 2024
Call File::sync_all when finalizing encoding

rust-lang#123352 (comment)

r? `@ghost`
bors added a commit to rust-lang-ci/rust that referenced this issue May 6, 2024
Add a footer in FileEncoder and check for it in MemDecoder

We have a few reports of ICEs due to decoding failures, where the fault does not lie with the compiler. The goal of this PR is to add some very lightweight and on-by-default validation to the compiler's outputs. If validation fails, we emit a fatal error for rmeta files in general that mentions the path that didn't load, and for incremental compilation artifacts we emit a verbose warning that tries to explain the situation and treat the artifacts as outdated.

The validation currently implemented here is very crude, and yet I think we have 11 ICE reports currently open (you can find them by searching issues for `1002111927320821928687967599834759150` which this simple validation would have detected. The structure of the code changes here should permit the addition of further validation code, such as a checksum, if it is merited. I would like to have code to detect corruption such as reported in rust-lang#124719, but I'm not yet sure how to do that efficiently, and this PR is already a good size.

The ICE reports I have in mind that this PR would have smoothed over are:
rust-lang#124469
rust-lang#123352
rust-lang#123376 [^1]
rust-lang#99763
rust-lang#93900.

---

[^1]: This one might be a compiler bug, but even if it is I think the workflow described is pushing the envelope of what we can support
bors added a commit to rust-lang-ci/rust that referenced this issue May 22, 2024
Add a footer in FileEncoder and check for it in MemDecoder

We have a few reports of ICEs due to decoding failures, where the fault does not lie with the compiler. The goal of this PR is to add some very lightweight and on-by-default validation to the compiler's outputs. If validation fails, we emit a fatal error for rmeta files in general that mentions the path that didn't load, and for incremental compilation artifacts we emit a verbose warning that tries to explain the situation and treat the artifacts as outdated.

The validation currently implemented here is very crude, and yet I think we have 11 ICE reports currently open (you can find them by searching issues for `1002111927320821928687967599834759150`) which this simple validation would have detected. The structure of the code changes here should permit the addition of further validation code, such as a checksum, if it is merited. I would like to have code to detect corruption such as reported in rust-lang#124719, but I'm not yet sure how to do that efficiently, and this PR is already a good size.

The ICE reports I have in mind that this PR would have smoothed over are:
rust-lang#124469
rust-lang#123352
rust-lang#123376 [^1]
rust-lang#99763
rust-lang#93900.

---

[^1]: This one might be a compiler bug, but even if it is I think the workflow described is pushing the envelope of what we can support. This issue is one of the reasons this warning still asks people to file an issue.
github-actions bot pushed a commit to rust-lang/miri that referenced this issue May 23, 2024
Add a footer in FileEncoder and check for it in MemDecoder

We have a few reports of ICEs due to decoding failures, where the fault does not lie with the compiler. The goal of this PR is to add some very lightweight and on-by-default validation to the compiler's outputs. If validation fails, we emit a fatal error for rmeta files in general that mentions the path that didn't load, and for incremental compilation artifacts we emit a verbose warning that tries to explain the situation and treat the artifacts as outdated.

The validation currently implemented here is very crude, and yet I think we have 11 ICE reports currently open (you can find them by searching issues for `1002111927320821928687967599834759150`) which this simple validation would have detected. The structure of the code changes here should permit the addition of further validation code, such as a checksum, if it is merited. I would like to have code to detect corruption such as reported in rust-lang/rust#124719, but I'm not yet sure how to do that efficiently, and this PR is already a good size.

The ICE reports I have in mind that this PR would have smoothed over are:
rust-lang/rust#124469
rust-lang/rust#123352
rust-lang/rust#123376 [^1]
rust-lang/rust#99763
rust-lang/rust#93900.

---

[^1]: This one might be a compiler bug, but even if it is I think the workflow described is pushing the envelope of what we can support. This issue is one of the reasons this warning still asks people to file an issue.
flip1995 pushed a commit to flip1995/rust-clippy that referenced this issue May 24, 2024
Add a footer in FileEncoder and check for it in MemDecoder

We have a few reports of ICEs due to decoding failures, where the fault does not lie with the compiler. The goal of this PR is to add some very lightweight and on-by-default validation to the compiler's outputs. If validation fails, we emit a fatal error for rmeta files in general that mentions the path that didn't load, and for incremental compilation artifacts we emit a verbose warning that tries to explain the situation and treat the artifacts as outdated.

The validation currently implemented here is very crude, and yet I think we have 11 ICE reports currently open (you can find them by searching issues for `1002111927320821928687967599834759150`) which this simple validation would have detected. The structure of the code changes here should permit the addition of further validation code, such as a checksum, if it is merited. I would like to have code to detect corruption such as reported in rust-lang/rust#124719, but I'm not yet sure how to do that efficiently, and this PR is already a good size.

The ICE reports I have in mind that this PR would have smoothed over are:
rust-lang/rust#124469
rust-lang/rust#123352
rust-lang/rust#123376 [^1]
rust-lang/rust#99763
rust-lang/rust#93900.

---

[^1]: This one might be a compiler bug, but even if it is I think the workflow described is pushing the envelope of what we can support. This issue is one of the reasons this warning still asks people to file an issue.
@saethlin
Copy link
Member

As far as I can tell, the ICE reported in this issue is caused by another process damaging rustc's build artifacts or by a buggy filesystem. That other process could be anything from an IDE running the compiler at the same time as a manual build to a find-and-replace that accidentally edits files in the target directory.

#124686 contains a mitigation for this situation; the compiler now has a crude way to detect and ignore damaged incremental compilation files. You will still get a warning about the damaged file, but the build should proceed normally. That change shipped in Rust 1.80 which released today, so I am closing this issue and a few others which as far as I can tell all have the same root cause. You shouldn't see this ICE anymore, but please file a new issue if you see something like it on 1.80 or later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-incr-comp Area: Incremental compilation C-bug Category: This is a bug. I-ICE Issue: The compiler panicked, giving an Internal Compilation Error (ICE) ❄️ T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

No branches or pull requests

5 participants