Panic due to auto-generated file diagnostics being incorrectly attributed to source files

[CeleritasCelery]

I have been struggling to find a way to reproduce this issue, and it has been plaguing me for a few weeks. While editing rust-analyzer will panic "randomly". It happens multiple times a day. I can't seem to find a good way to make the issue reproducible, but here is the backtrace below. I have uninstalled and reinstalled rust-analyzer multiple times, and used cargo clean to clean up the cache, but nothing seems to fix it.

Panic context:
> fetch_native_diagnostics

thread 'Worker' panicked at 'invalid offset', /Users/brew/Library/Caches/Homebrew/cargo_cache/registry/src/index.crates.io-6f17d22bba15001f/line-index-0.1.0-pre.1/src/lib.rs:148:35
stack backtrace:
   0:        0x10332c8d4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h782e083efa4700b5
   1:        0x103342910 - core::fmt::write::hdcd44492977669f9
   2:        0x103319590 - std::io::Write::write_fmt::h7f32f27b72970ad5
   3:        0x10332c6dc - std::sys_common::backtrace::print::h6868989d298a5d93
   4:        0x10331a268 - std::panicking::default_hook::{{closure}}::he58f0ab435f26824
   5:        0x103319fd4 - std::panicking::default_hook::h0f0ccb5a1a0c4466
   6:        0x1032acbc4 - std::thread::local::LocalKey<T>::with::h8a04591509e74c6c
   7:        0x10331a894 - std::panicking::rust_panic_with_hook::hf392a622566eb9b6
   8:        0x10332d23c - std::panicking::begin_panic_handler::{{closure}}::h07b7cf65243520e5
   9:        0x10332c9c8 - std::sys_common::backtrace::__rust_end_short_backtrace::h4972088065ba8a0d
  10:        0x10331a410 - _rust_begin_unwind
  11:        0x10340c2ec - core::panicking::panic_fmt::hc67fbbe16e2d8abd
  12:        0x10340c208 - core::option::expect_failed::h287fa49622669d1e
  13:        0x102b92d94 - line_index::LineIndex::line_col::h9da28fa25839dd78
  14:        0x102542d60 - rust_analyzer::lsp::to_proto::range::h6353167a0a495419
  15:        0x10259a4c8 - rust_analyzer::diagnostics::fetch_native_diagnostics::{{closure}}::{{closure}}::hecf8a81f0f825fad
  16:        0x1025928fc - <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::fold::h1c39745215b5ec36
  17:        0x1023decc8 - alloc::vec::in_place_collect::<impl alloc::vec::spec_from_iter::SpecFromIter<T,I> for alloc::vec::Vec<T>>::from_iter::hab9256ef809ab5d9
  18:        0x1024c3c74 - core::ops::function::impls::<impl core::ops::function::FnMut<A> for &mut F>::call_mut::h275bc8a58b943c81
  19:        0x1023de7e8 - alloc::vec::in_place_collect::<impl alloc::vec::spec_from_iter::SpecFromIter<T,I> for alloc::vec::Vec<T>>::from_iter::ha439e318cacb3078
  20:        0x10254e588 - rust_analyzer::diagnostics::fetch_native_diagnostics::hb716dbe6f0d7efb0
  21:        0x1024427d4 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h4f72f34972931faf
  22:        0x1032b3860 - std::sys_common::backtrace::__rust_begin_short_backtrace::h41920f26e3f6a8de
  23:        0x1032b4c44 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hc0cb8167e3f64f0d
  24:        0x103310640 - std::sys::unix::thread::Thread::new::thread_start::hfdb57e802740a0f8
  25:        0x1909bffa8 - __pthread_joiner_wake
thread 'LspServer' panicked at 'called `Result::unwrap()` on an `Err` value: "SendError(..)"', /private/tmp/rust-analyzer-20231023-5820-ffnuam/crates/stdx/src/thread/pool.rs:86:35
stack backtrace:
   0:        0x10332c8d4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h782e083efa4700b5
   1:        0x103342910 - core::fmt::write::hdcd44492977669f9
   2:        0x103319590 - std::io::Write::write_fmt::h7f32f27b72970ad5
   3:        0x10332c6dc - std::sys_common::backtrace::print::h6868989d298a5d93
   4:        0x10331a268 - std::panicking::default_hook::{{closure}}::he58f0ab435f26824
   5:        0x103319fd4 - std::panicking::default_hook::h0f0ccb5a1a0c4466
   6:        0x1032acbc4 - std::thread::local::LocalKey<T>::with::h8a04591509e74c6c
   7:        0x10331a894 - std::panicking::rust_panic_with_hook::hf392a622566eb9b6
   8:        0x10332d23c - std::panicking::begin_panic_handler::{{closure}}::h07b7cf65243520e5
   9:        0x10332c9c8 - std::sys_common::backtrace::__rust_end_short_backtrace::h4972088065ba8a0d
  10:        0x10331a410 - _rust_begin_unwind
  11:        0x10340c2ec - core::panicking::panic_fmt::hc67fbbe16e2d8abd
  12:        0x10340c2b8 - core::result::unwrap_failed::had2324ff96cef11b
  13:        0x102450e40 - stdx::thread::pool::Pool::spawn::h676b672002511263
  14:        0x102537588 - rust_analyzer::main_loop::<impl rust_analyzer::global_state::GlobalState>::run::h85cccb15a6f57591
  15:        0x1025a1270 - rust_analyzer::main_loop::main_loop::hced9114fa7d7bda5
  16:        0x102370d6c - rust_analyzer::run_server::h77f7ea5391c0891c
  17:        0x1023a37b0 - std::sys_common::backtrace::__rust_begin_short_backtrace::h6f9b411514b7c434
  18:        0x102396b94 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h139b1f5b35fd3948
  19:        0x103310640 - std::sys::unix::thread::Thread::new::thread_start::hfdb57e802740a0f8
  20:        0x1909bffa8 - __pthread_joiner_wake
thread '<unnamed>' panicked at 'receiver was dropped, failed to send a message: "SendError(..)"', /Users/brew/Library/Caches/Homebrew/cargo_cache/registry/src/index.crates.io-6f17d22bba15001f/lsp-server-0.7.4/src/stdio.rs:29:37
stack backtrace:
   0:        0x10332c8d4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h782e083efa4700b5
   1:        0x103342910 - core::fmt::write::hdcd44492977669f9
   2:        0x103319590 - std::io::Write::write_fmt::h7f32f27b72970ad5
   3:        0x10332c6dc - std::sys_common::backtrace::print::h6868989d298a5d93
   4:        0x10331a268 - std::panicking::default_hook::{{closure}}::he58f0ab435f26824
   5:        0x103319fd4 - std::panicking::default_hook::h0f0ccb5a1a0c4466
   6:        0x1032acbc4 - std::thread::local::LocalKey<T>::with::h8a04591509e74c6c
   7:        0x10331a894 - std::panicking::rust_panic_with_hook::hf392a622566eb9b6
   8:        0x10332d23c - std::panicking::begin_panic_handler::{{closure}}::h07b7cf65243520e5
   9:        0x10332c9c8 - std::sys_common::backtrace::__rust_end_short_backtrace::h4972088065ba8a0d
  10:        0x10331a410 - _rust_begin_unwind
  11:        0x10340c2ec - core::panicking::panic_fmt::hc67fbbe16e2d8abd
  12:        0x10340c2b8 - core::result::unwrap_failed::had2324ff96cef11b
  13:        0x1032cdfc8 - std::sys_common::backtrace::__rust_begin_short_backtrace::hffb21713b0df89b6
  14:        0x1032beddc - core::ops::function::FnOnce::call_once{{vtable.shim}}::h9ac33306162a5bbb
  15:        0x103310640 - std::sys::unix::thread::Thread::new::thread_start::hfdb57e802740a0f8
  16:        0x1909bffa8 - __pthread_joiner_wake
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Any { .. }', /Users/brew/Library/Caches/Homebrew/cargo_cache/registry/src/index.crates.io-6f17d22bba15001f/jod-thread-0.1.2/src/lib.rs:33:22
stack backtrace:
   0:        0x10332c8d4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h782e083efa4700b5
   1:        0x103342910 - core::fmt::write::hdcd44492977669f9
   2:        0x103319590 - std::io::Write::write_fmt::h7f32f27b72970ad5
   3:        0x10332c6dc - std::sys_common::backtrace::print::h6868989d298a5d93
   4:        0x10331a268 - std::panicking::default_hook::{{closure}}::he58f0ab435f26824
   5:        0x103319fd4 - std::panicking::default_hook::h0f0ccb5a1a0c4466
   6:        0x1032acbc4 - std::thread::local::LocalKey<T>::with::h8a04591509e74c6c
   7:        0x10331a894 - std::panicking::rust_panic_with_hook::hf392a622566eb9b6
   8:        0x10332d23c - std::panicking::begin_panic_handler::{{closure}}::h07b7cf65243520e5
   9:        0x10332c9c8 - std::sys_common::backtrace::__rust_end_short_backtrace::h4972088065ba8a0d
  10:        0x10331a410 - _rust_begin_unwind
  11:        0x10340c2ec - core::panicking::panic_fmt::hc67fbbe16e2d8abd
  12:        0x10340c2b8 - core::result::unwrap_failed::had2324ff96cef11b
  13:        0x1023a3c38 - jod_thread::JoinHandle<T>::join::h27d24a5684560a1c
  14:        0x1023a3b30 - rust_analyzer::with_extra_thread::hb6ebbb3b917cb384
  15:        0x10236fd1c - rust_analyzer::main::h63ad5db90324360b
  16:        0x1023a37c4 - std::sys_common::backtrace::__rust_begin_short_backtrace::hd3d0d914a4d9e9c3
  17:        0x1023961fc - std::rt::lang_start::{{closure}}::h141fca22b63bfc08
  18:        0x103318aa0 - std::rt::lang_start_internal::h3e84e2f86cd60d8c
  19:        0x102371558 - _main

Process rust-analyzer stderr finished

rust-analyzer version: rust-analyzer 0.0.0 (1087295 2023-10-22)

rustc version: rustc 1.74.0-beta.1 (b5c050feb 2023-10-03)

relevant settings: default settings

[lnicola]

Are you using non-ASCII characters in your code? There's currently a bug related to that (#15273).

[CeleritasCelery]

I did have one non-ascii character in a string literal. However removing that and adding #![deny(clippy::non_ascii_literal)] has not resolved this issue (no other non-ascii chars).

I have found a way to reproduce the issue, but it is not a minimal reproduction. I built rust-analyzer from source, and I was able to see that it is triggering on this line.

    pub fn line_col(&self, offset: TextSize) -> LineCol {
        self.try_line_col(offset).expect("invalid offset")
    }

However what I am confused about is that the stderr from RA when it crashes points to this file:

/Users/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/line-index-0.1.0-pre.1/src/lib.rs:148:35

That file does not match what I see in my clone of RA (which is latest). I would have expected if I built from source that source code cache should match what is in the repo I am building from no? It looks like the src cache for that file is pointing to 60056b8.

[lnicola]

It's not just about literals. Even comments can trigger it.

The line-index crate is a crates.io dependency, you need to replace it with a path dependency. See

rust-analyzer/Cargo.toml

Line 88 in 1087295

line-index = { version = "0.1.0-pre.1" }

.

[CeleritasCelery]

I wasn't very clear, but I did check for all non-ascii characters (via rg "[\x80-\xFF]" src).

I added some prints and I can see that the issue is this check

rust-analyzer/lib/line-index/src/lib.rs

Line 119 in 46c395d

if offset > self.len {

The values when it fails is offset: 23243, self.len: 16429. The len and number of lines matches what is in my file, but obviously the offset is way off. I need to determine where that is coming from. FWIW, the code that triggers this is when typing a macro, not sure if that is relevant.

What is the best way to go about debugging this? So far I have just been adding eprintln statements to the source and rebuilding, but that is a slow feedback loop.

[lnicola]

I'm not that familiar with ripgrep, is it Unicode by default? That regex might miss Unicode chars.

I don't know a good way to debug these, and macros can cause issues on their own, but those usually look a bit different. I generally try to minimize the code, but the line index is very sensitive to your edits.

[CeleritasCelery]

I did some more debugging and traced the issue back to this

rust-analyzer/crates/rust-analyzer/src/diagnostics.rs

Line 126 in 46c395d

pub(crate) fn fetch_native_diagnostics(

I am receiving this diagnostic while editing.

Diagnostic {
    code: RustcLint("non_upper_case_globals"),
    message: "Constant `INTERNAL__DEFINE_UNINITIALIZED_VARIABLE` should have UPPER_SNAKE_CASE name, e.g. `INTERNAL_DEFINE_UNINITIALIZED_VARIABLE`",
    range: 23243..23282,
    severity: Warning,
    unused: false,
    experimental: false,
    fixes: None,
    main_node: Some(InFile {
        file_id: MacroFile(MacroFile { macro_call_id: MacroCallId(6976) }),
        value: NAME@20598..20637 })
}

The range is out of bounds for the file being editing. In fact this diagnostic does not even apply to that file. This particular diagnostic belongs to an auto-generated file created by build.rs. In that file, the range 23243..23383 is correct. So it looks like RA is fetching the diagnostics for the auto-generated file and applying them to "normal" source files. Maybe because that is files is not part of the normal source tree?

I don't understand enough about RA's architecture to determine why this is happening, but hopefully this will help you point me in the right direction.

[lnicola]

Yeah, we sometimes mix up macro expansions with the files containing the macro calls. #15807 is an example of that.

[lnicola]

Can you try to disable the diagnostic (there a setting) or disable check on save and see if it still crashes?

Or even fix the warning if possible.

[CeleritasCelery]

So the macro call was a red herring, I can trigger this without any macros. In fact just typing let x = "hello world" will trigger it. All that I need is to be editing a file that is smaller then the one generated by build.rs.

[CeleritasCelery]

Disabling the diagnostic "fixed" the issue. That is good to confirm. But I would like to find a way to fix RA.

[lnicola]

Can you enable it back and disable rust-analyzer.checkOnSave, then restart Code? Some diagnostics are from us and some from rustc and it's not always obvious which is which (yes, I noticed the RustcLint there).

[CeleritasCelery]

re-enabling the diagnostic and disabling rust-analyzer.checkOnSave still shows the issue.