Verify that spans point to char boundaries #106192

Noratrieb · 2022-12-27T20:22:47Z

This makes invalid spans a lot easier to debug. A quick and unscientific benchmarking test revealed that the performance impact of this is small at most.

I added this for debugging #106191

This PR also fixes a small bug that was found by the debug assertions.

rustbot · 2022-12-27T20:22:52Z

r? @cjgillot

(rustbot has picked a reviewer for you, use r? to override)

anden3 · 2023-04-13T19:32:27Z

Hello @Nilstrieb! What's the status of this PR?

Noratrieb · 2023-04-13T19:40:13Z

i don't know
I'd like to fix it but also that seems annoying and not too important. If you want me to close it I'll open an issue and close it. I probably won't work on it in the near future.

anden3 · 2023-04-13T19:43:31Z

It's alright, just wanted to make sure you hadn't forgotten about it!
I'll probably re-ping you in a few months or so if that's fine with you :)

This makes invalid spans a lot easier to debug. A quick and unscientific benchmarking test revealed that the performance impact of this is small at most.

When the span is empty, it doesn't really have a first character. Even worse, when it's empty at the end of the file, adding a byte offset will make it be out of bounds. So we just return the empty span in these cases.

Noratrieb · 2024-01-12T21:54:07Z

This is a potentially expensive debug assertion, so I benchmarked x test tests/ui (just running the tests)
with this PR: 130s
master: 123s

could be noise, but it doesn't look like a huge slowdown

though i'm still not sure whether I'll ever fix all the issues and merge this

rust-log-analyzer · 2024-01-12T22:28:03Z

The job x86_64-gnu-llvm-16 failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

GITHUB_ENV=/home/runner/work/_temp/_runner_file_commands/set_env_274f9b41-7b14-46b3-bb15-9908b2c9b602
GITHUB_EVENT_NAME=pull_request
GITHUB_EVENT_PATH=/home/runner/work/_temp/_github_workflow/event.json
GITHUB_GRAPHQL_URL=https://api.github.com/graphql
GITHUB_HEAD_REF=verify-span-correctness
GITHUB_JOB=pr
GITHUB_PATH=/home/runner/work/_temp/_runner_file_commands/add_path_274f9b41-7b14-46b3-bb15-9908b2c9b602
GITHUB_REF=refs/pull/106192/merge
GITHUB_REF_NAME=106192/merge
GITHUB_REF_PROTECTED=false
---
Built container sha256:9c3c93a371e5aed5c18185b24f130d95d5140dbd72a9b325e7b6b49e521a4faa
Looks like docker image is the same as before, not uploading
https://ci-caches.rust-lang.org/docker/7ebc15c01a233894034d277c8cce4e949f4e7791f66b4727c8fb6e058a0b8171d6152e1441d677cef0653843ceeee469c097b8699b2bb74249e674f6aa1a8813
sha256:9c3c93a371e5aed5c18185b24f130d95d5140dbd72a9b325e7b6b49e521a4faa
Setting extra environment values for docker:  --env ENABLE_GCC_CODEGEN=1 --env GCC_EXEC_PREFIX=/usr/lib/gcc/
[CI_JOB_NAME=x86_64-gnu-llvm-16]
##[group]Clock drift check
  local time: Fri Jan 12 21:56:11 UTC 2024
  network time: Fri, 12 Jan 2024 21:56:11 GMT
  network time: Fri, 12 Jan 2024 21:56:11 GMT
##[endgroup]
sccache: Starting the server...
##[group]Configure the build
configure: processing command line
configure: 
configure: build.configure-args := ['--build=x86_64-unknown-linux-gnu', '--llvm-root=/usr/lib/llvm-16', '--enable-llvm-link-shared', '--set', 'rust.thin-lto-import-instr-limit=10', '--set', 'change-id=99999999', '--enable-verbose-configure', '--enable-sccache', '--disable-manage-submodules', '--enable-locked-deps', '--enable-cargo-native-static', '--set', 'rust.codegen-units-std=1', '--set', 'dist.compression-profile=balanced', '--dist-compression-formats=xz', '--set', 'build.optimized-compiler-builtins', '--disable-dist-src', '--release-channel=nightly', '--enable-debug-assertions', '--enable-overflow-checks', '--enable-llvm-assertions', '--set', 'rust.verify-llvm-ir', '--set', 'rust.codegen-backends=llvm,cranelift,gcc', '--set', 'llvm.static-libstdcpp', '--enable-new-symbol-mangling']
configure: target.x86_64-unknown-linux-gnu.llvm-config := /usr/lib/llvm-16/bin/llvm-config
configure: llvm.link-shared     := True
configure: rust.thin-lto-import-instr-limit := 10
configure: change-id            := 99999999
---
##[endgroup]
Testing GCC stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
   Compiling y v0.1.0 (/checkout/compiler/rustc_codegen_gcc/build_system)
    Finished release [optimized] target(s) in 1.36s
     Running `/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-codegen/x86_64-unknown-linux-gnu/release/y test --use-system-gcc --use-backend gcc --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/stage1-tools/cg_gcc --release --no-default-features --mini-tests --std-tests`
Using system GCC
Using system GCC
[BUILD] example
[AOT] mini_core_hello_world
/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-tools/cg_gcc/mini_core_hello_world
abc
---
##[group]Documenting stage2 book redirect pages (x86_64-unknown-linux-gnu)
thread 'rustc' panicked at compiler/rustc_span/src/source_map.rs:987:9:
attempt to subtract with overflow
stack backtrace:
   0:     0x7fd7451645f9 - <std[3adfa28e23c93245]::sys_common::backtrace::_print::DisplayBacktrace as core[56dfe42b576a4ec4]::fmt::Display>::fmt
   1:     0x7fd7451be18f - core[56dfe42b576a4ec4]::fmt::write
   2:     0x7fd745158e45 - <std[3adfa28e23c93245]::sys::unix::stdio::Stderr as std[3adfa28e23c93245]::io::Write>::write_fmt
   3:     0x7fd7451643d5 - std[3adfa28e23c93245]::sys_common::backtrace::print
   4:     0x7fd74516770a - std[3adfa28e23c93245]::panicking::default_hook::{closure#1}
   5:     0x7fd74516740d - std[3adfa28e23c93245]::panicking::default_hook
   6:     0x7fd745dece17 - <alloc[4c32687bb71ada8a]::boxed::Box<rustc_driver_impl[945e6cd7b9f88e04]::install_ice_hook::{closure#0}> as core[56dfe42b576a4ec4]::ops::function::Fn<(&dyn for<'a, 'b> core[56dfe42b576a4ec4]::ops::function::Fn<(&'a core[56dfe42b576a4ec4]::panic::panic_info::PanicInfo<'b>,), Output = ()> + core[56dfe42b576a4ec4]::marker::Send + core[56dfe42b576a4ec4]::marker::Sync, &core[56dfe42b576a4ec4]::panic::panic_info::PanicInfo)>>::call
   7:     0x7fd745167dc4 - std[3adfa28e23c93245]::panicking::rust_panic_with_hook
   8:     0x7fd745167b19 - std[3adfa28e23c93245]::panicking::begin_panic_handler::{closure#0}
   9:     0x7fd745164b86 - std[3adfa28e23c93245]::sys_common::backtrace::__rust_end_short_backtrace::<std[3adfa28e23c93245]::panicking::begin_panic_handler::{closure#0}, !>
  11:     0x7fd74511ea95 - core[56dfe42b576a4ec4]::panicking::panic_fmt
  12:     0x7fd74511eb53 - core[56dfe42b576a4ec4]::panicking::panic
  13:     0x7fd7491a522e - <rustc_span[c41625dfce96a22b]::source_map::SourceMap>::lookup_source_file_idx
  14:     0x7fd7491a5023 - <rustc_span[c41625dfce96a22b]::source_map::SourceMap>::lookup_byte_offset
  14:     0x7fd7491a5023 - <rustc_span[c41625dfce96a22b]::source_map::SourceMap>::lookup_byte_offset
  15:     0x7fd748a6ebc5 - <scoped_tls[a2788e8dfce51fe3]::ScopedKey<rustc_span[c41625dfce96a22b]::SessionGlobals>>::with::<<rustc_span[c41625dfce96a22b]::span_encoding::Span>::new::{closure#0}, ()>
  16:     0x7fd748ab8920 - <rustc_parse[419d5da128d8d697]::lexer::StringReader>::mk_sp
  17:     0x7fd748aba08a - <rustc_parse[419d5da128d8d697]::lexer::StringReader>::next_token
  18:     0x7fd748ac6ca2 - <rustc_parse[419d5da128d8d697]::lexer::tokentrees::TokenTreesReader>::parse_token_trees
  19:     0x7fd748ac6b58 - <rustc_parse[419d5da128d8d697]::lexer::tokentrees::TokenTreesReader>::parse_all_token_trees
  20:     0x7fd748ab8472 - rustc_parse[419d5da128d8d697]::lexer::parse_token_trees
  21:     0x7fd748ac8f11 - rustc_parse[419d5da128d8d697]::maybe_file_to_stream
  22:     0x7fd748ac8beb - rustc_parse[419d5da128d8d697]::maybe_source_file_to_parser
  23:     0x7fd748ac892e - rustc_parse[419d5da128d8d697]::maybe_new_parser_from_source_str
  24:     0x560d191df36e - <scoped_tls[a2788e8dfce51fe3]::ScopedKey<rustc_span[c41625dfce96a22b]::SessionGlobals>>::with::<rustdoc[83686be7d147ba14]::doctest::make_test::{closure#0}::{closure#0}, (bool, bool, bool)>
  25:     0x560d191e9f1f - std[3adfa28e23c93245]::panicking::try::<(bool, bool, bool), core[56dfe42b576a4ec4]::panic::unwind_safe::AssertUnwindSafe<rustdoc[83686be7d147ba14]::doctest::make_test::{closure#0}>>
  26:     0x560d194d36ff - rustdoc[83686be7d147ba14]::doctest::make_test
  27:     0x560d19487684 - <rustdoc[83686be7d147ba14]::html::markdown::CodeBlocks<rustdoc[83686be7d147ba14]::html::markdown::TableWrapper<rustdoc[83686be7d147ba14]::html::markdown::LinkReplacer<core[56dfe42b576a4ec4]::iter::adapters::map::Map<rustdoc[83686be7d147ba14]::html::markdown::Footnotes<rustdoc[83686be7d147ba14]::html::markdown::HeadingLinks<pulldown_cmark[4299e9c5aefec2bc]::parse::OffsetIter>>, <rustdoc[83686be7d147ba14]::html::markdown::Markdown>::into_string::{closure#1}>>>> as core[56dfe42b576a4ec4]::iter::traits::iterator::Iterator>::next
  28:     0x560d1949d9a5 - pulldown_cmark[4299e9c5aefec2bc]::html::push_html::<rustdoc[83686be7d147ba14]::html::markdown::CodeBlocks<rustdoc[83686be7d147ba14]::html::markdown::TableWrapper<rustdoc[83686be7d147ba14]::html::markdown::LinkReplacer<core[56dfe42b576a4ec4]::iter::adapters::map::Map<rustdoc[83686be7d147ba14]::html::markdown::Footnotes<rustdoc[83686be7d147ba14]::html::markdown::HeadingLinks<pulldown_cmark[4299e9c5aefec2bc]::parse::OffsetIter>>, <rustdoc[83686be7d147ba14]::html::markdown::Markdown>::into_string::{closure#1}>>>>>
  29:     0x560d19493c35 - <rustdoc[83686be7d147ba14]::html::markdown::Markdown>::into_string
  30:     0x560d194daf0d - rustdoc[83686be7d147ba14]::markdown::render::<&std[3adfa28e23c93245]::path::PathBuf>
  31:     0x560d192169e7 - rustc_interface[24dc5d0625193760]::interface::run_compiler::<core[56dfe42b576a4ec4]::result::Result<(), alloc[4c32687bb71ada8a]::string::String>, rustdoc[83686be7d147ba14]::main_args::{closure#0}>::{closure#0}
  32:     0x560d191db553 - rustc_span[c41625dfce96a22b]::create_session_globals_then::<core[56dfe42b576a4ec4]::result::Result<(), alloc[4c32687bb71ada8a]::string::String>, rustc_interface[24dc5d0625193760]::util::run_in_thread_pool_with_globals<rustc_interface[24dc5d0625193760]::interface::run_compiler<core[56dfe42b576a4ec4]::result::Result<(), alloc[4c32687bb71ada8a]::string::String>, rustdoc[83686be7d147ba14]::main_args::{closure#0}>::{closure#0}, core[56dfe42b576a4ec4]::result::Result<(), alloc[4c32687bb71ada8a]::string::String>>::{closure#0}>
  33:     0x560d193a4122 - std[3adfa28e23c93245]::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface[24dc5d0625193760]::util::run_in_thread_with_globals<rustc_interface[24dc5d0625193760]::util::run_in_thread_pool_with_globals<rustc_interface[24dc5d0625193760]::interface::run_compiler<core[56dfe42b576a4ec4]::result::Result<(), alloc[4c32687bb71ada8a]::string::String>, rustdoc[83686be7d147ba14]::main_args::{closure#0}>::{closure#0}, core[56dfe42b576a4ec4]::result::Result<(), alloc[4c32687bb71ada8a]::string::String>>::{closure#0}, core[56dfe42b576a4ec4]::result::Result<(), alloc[4c32687bb71ada8a]::string::String>>::{closure#0}::{closure#0}, core[56dfe42b576a4ec4]::result::Result<(), alloc[4c32687bb71ada8a]::string::String>>
  34:     0x560d192186cd - <<std[3adfa28e23c93245]::thread::Builder>::spawn_unchecked_<rustc_interface[24dc5d0625193760]::util::run_in_thread_with_globals<rustc_interface[24dc5d0625193760]::util::run_in_thread_pool_with_globals<rustc_interface[24dc5d0625193760]::interface::run_compiler<core[56dfe42b576a4ec4]::result::Result<(), alloc[4c32687bb71ada8a]::string::String>, rustdoc[83686be7d147ba14]::main_args::{closure#0}>::{closure#0}, core[56dfe42b576a4ec4]::result::Result<(), alloc[4c32687bb71ada8a]::string::String>>::{closure#0}, core[56dfe42b576a4ec4]::result::Result<(), alloc[4c32687bb71ada8a]::string::String>>::{closure#0}::{closure#0}, core[56dfe42b576a4ec4]::result::Result<(), alloc[4c32687bb71ada8a]::string::String>>::{closure#1} as core[56dfe42b576a4ec4]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  35:     0x7fd74517394e - <std[3adfa28e23c93245]::sys::unix::thread::Thread>::new::thread_start
  37:     0x7fd744ee00d0 - <unknown>
  38:                0x0 - <unknown>

error: the compiler unexpectedly panicked. this is a bug.
error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-rustdoc&template=ice.md

note: please attach the file at `/checkout/obj/rustc-ice-2024-01-12T22_26_31-254411.txt` to your bug report

note: compiler flags: -Z normalize-docs -Z unstable-options
query stack during panic:
end of query stack
Build completed unsuccessfully in 0:29:44
  local time: Fri Jan 12 22:26:31 UTC 2024

rustbot assigned cjgillot Dec 27, 2022

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Dec 27, 2022