ICE: rmeta/decoder.rs:967:9: missing def_span for DefId(117:0 ~ test[50b7])

[edmorley]

I just encountered an ICE in CI during cargo test with Rust 1.8.4 stable on a Linux ARM GitHub Actions runner, after merging a PR to main. (CI passed on the PR but failed on the main run post-merge.)

Code

The failure was on this commit:
https://github.com/heroku/buildpacks-python/tree/aef5e6931d049d27a5c52437ecb1dfa6f9ff2f41

Meta

rustc --version --verbose:

rustc 1.84.0 (9fc6b4312 2025-01-07)
binary: rustc
commit-hash: 9fc6b43126469e3858e2fe86cafb4f0fd5068869
commit-date: 2025-01-07
host: aarch64-unknown-linux-gnu
release: 1.84.0
LLVM version: 19.1.5

Error output

The full CI run can be seen here:
https://github.com/heroku/buildpacks-python/actions/runs/12934086542/job/36074243839#step:10:13

The GitHub Actions workflow is defined here:
https://github.com/heroku/buildpacks-python/blob/aef5e6931d049d27a5c52437ecb1dfa6f9ff2f41/.github/workflows/ci.yml#L43-L86

$ cargo test --locked -- --ignored --test-threads $(($(nproc)+1))
    Updating crates.io index
   Compiling python-buildpack v0.0.0 (/home/runner/work/buildpacks-python/buildpacks-python)
error: internal compiler error: compiler/rustc_metadata/src/rmeta/decoder.rs:967:9: missing `def_span` for DefId(117:0 ~ test[50b7])

thread 'rustc' panicked at compiler/rustc_metadata/src/rmeta/decoder.rs:967:9:
Box<dyn Any>
stack backtrace:
   0:     0xffae6a41186c - std::backtrace_rs::backtrace::libunwind::trace::h41252309c76992b5
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
   1:     0xffae6a41186c - std::backtrace_rs::backtrace::trace_unsynchronized::hf225d125cca37a81
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0xffae6a41186c - std::sys::backtrace::_print_fmt::hf329c51b3772454a
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/sys/backtrace.rs:66:9
   3:     0xffae6a41186c - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h00b09a4bf39b2d85
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/sys/backtrace.rs:39:26
   4:     0xffae6a45e62c - core::fmt::rt::Argument::fmt::heddda1cce7046e46
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/fmt/rt.rs:177:76
   5:     0xffae6a45e62c - core::fmt::write::h7ceb4b8480f8c249
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/fmt/mod.rs:1189:21
   6:     0xffae6a4058e0 - std::io::Write::write_fmt::h85dd460e8a217df7
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/io/mod.rs:1884:15
   7:     0xffae6a411720 - std::sys::backtrace::BacktraceLock::print::h9a7ef9a5429d1e7e
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/sys/backtrace.rs:42:9
   8:     0xffae6a413a60 - std::panicking::default_hook::{{closure}}::h0c27709af3f8d4c6
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/panicking.rs:268:22
   9:     0xffae6a4138a8 - std::panicking::default_hook::h896b14ac97871c98
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/panicking.rs:295:9
  10:     0xffae64235e7c - <alloc[b23c587ff69f26b0]::boxed::Box<rustc_driver_impl[233ea3aee64cd688]::install_ice_hook::{closure#0}> as core[2b1f2bbd3b605c]::ops::function::Fn<(&dyn for<'a, 'b> core[2b1f2bbd3b605c]::ops::function::Fn<(&'a std[a32bae77f370052a]::panic::PanicHookInfo<'b>,), Output = ()> + core[2b1f2bbd3b605c]::marker::Sync + core[2b1f2bbd3b605c]::marker::Send, &std[a32bae77f370052a]::panic::PanicHookInfo)>>::call
  11:     0xffae6a4141d0 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::he48809e66f217622
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/boxed.rs:1986:9
  12:     0xffae6a4141d0 - std::panicking::rust_panic_with_hook::h660add0bcc8a80ea
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/panicking.rs:809:13
  13:     0xffae6a099ad4 - std[a32bae77f370052a]::panicking::begin_panic::<rustc_errors[cc4532cbff5f0758]::ExplicitBug>::{closure#0}
  14:     0xffae6a0965b8 - std[a32bae77f370052a]::sys::backtrace::__rust_end_short_backtrace::<std[a32bae77f370052a]::panicking::begin_panic<rustc_errors[cc4532cbff5f0758]::ExplicitBug>::{closure#0}, !>
  15:     0xffae640ce358 - std[a32bae77f370052a]::panicking::begin_panic::<rustc_errors[cc4532cbff5f0758]::ExplicitBug>
  16:     0xffae6a086468 - <rustc_errors[cc4532cbff5f0758]::diagnostic::BugAbort as rustc_errors[cc4532cbff5f0758]::diagnostic::EmissionGuarantee>::emit_producing_guarantee
  17:     0xffae69d8f1a0 - rustc_middle[2c6500b4474def4b]::util::bug::opt_span_bug_fmt::<rustc_span[d6938c8799c78ecd]::span_encoding::Span>::{closure#0}
  18:     0xffae69d8dcc8 - rustc_middle[2c6500b4474def4b]::ty::context::tls::with_opt::<rustc_middle[2c6500b4474def4b]::util::bug::opt_span_bug_fmt<rustc_span[d6938c8799c78ecd]::span_encoding::Span>::{closure#0}, !>::{closure#0}
  19:     0xffae69d8dc98 - rustc_middle[2c6500b4474def4b]::ty::context::tls::with_context_opt::<rustc_middle[2c6500b4474def4b]::ty::context::tls::with_opt<rustc_middle[2c6500b4474def4b]::util::bug::opt_span_bug_fmt<rustc_span[d6938c8799c78ecd]::span_encoding::Span>::{closure#0}, !>::{closure#0}, !>
  20:     0xffae640b47f8 - rustc_middle[2c6500b4474def4b]::util::bug::bug_fmt
  21:     0xffae6983a8e4 - <rustc_metadata[bc7dd363cf28a80b]::creader::CrateMetadataRef>::missing
  22:     0xffae6983ae00 - <rustc_metadata[bc7dd363cf28a80b]::creader::CrateMetadataRef>::get_span
  23:     0xffae689aebe0 - <rustc_resolve[803f36c130519f4a]::Resolver>::def_span
  24:     0xffae6897a2e8 - <rustc_resolve[803f36c130519f4a]::Resolver>::get_module
  25:     0xffae6897a034 - <rustc_resolve[803f36c130519f4a]::Resolver>::expect_module
  26:     0xffae68a3712c - <rustc_resolve[803f36c130519f4a]::build_reduced_graph::BuildReducedGraphVisitor as rustc_ast[ba5aba888c7ba019]::visit::Visitor>::visit_item
  27:     0xffae689e99cc - <rustc_expand[448f8608944c7a44]::expand::AstFragment>::visit_with::<rustc_resolve[803f36c130519f4a]::build_reduced_graph::BuildReducedGraphVisitor>
  28:     0xffae6899cd74 - <rustc_resolve[803f36c130519f4a]::Resolver as rustc_expand[448f8608944c7a44]::base::ResolverExpand>::visit_ast_fragment_with_placeholders
  29:     0xffae69907048 - <rustc_expand[448f8608944c7a44]::expand::MacroExpander>::collect_invocations
  30:     0xffae69905330 - <rustc_expand[448f8608944c7a44]::expand::MacroExpander>::fully_expand_fragment
  31:     0xffae6990298c - <rustc_expand[448f8608944c7a44]::expand::MacroExpander>::expand_crate
  32:     0xffae643bf2c8 - <rustc_session[8fe50856523cfc81]::session::Session>::time::<rustc_ast[ba5aba888c7ba019]::ast::Crate, rustc_interface[45f6ee680972b70c]::passes::configure_and_expand::{closure#1}>
  33:     0xffae6442d8cc - rustc_interface[45f6ee680972b70c]::passes::resolver_for_lowering_raw
  34:     0xffae68d0086c - rustc_query_impl[b3a5340e321ea8a9]::plumbing::__rust_begin_short_backtrace::<rustc_query_impl[b3a5340e321ea8a9]::query_impl::resolver_for_lowering_raw::dynamic_query::{closure#2}::{closure#0}, rustc_middle[2c6500b4474def4b]::query::erase::Erased<[u8; 16usize]>>
  35:     0xffae68d53d70 - <rustc_query_impl[b3a5340e321ea8a9]::query_impl::resolver_for_lowering_raw::dynamic_query::{closure#2} as core[2b1f2bbd3b605c]::ops::function::FnOnce<(rustc_middle[2c6500b4474def4b]::ty::context::TyCtxt, ())>>::call_once
  36:     0xffae68f782c4 - rustc_query_system[4379a6dd4de40329]::query::plumbing::try_execute_query::<rustc_query_impl[b3a5340e321ea8a9]::DynamicConfig<rustc_query_system[4379a6dd4de40329]::query::caches::SingleCache<rustc_middle[2c6500b4474def4b]::query::erase::Erased<[u8; 16usize]>>, false, false, false>, rustc_query_impl[b3a5340e321ea8a9]::plumbing::QueryCtxt, false>
  37:     0xffae68dffd58 - rustc_query_impl[b3a5340e321ea8a9]::query_impl::resolver_for_lowering_raw::get_query_non_incr::__rust_end_short_backtrace
  38:     0xffae69d42454 - <rustc_middle[2c6500b4474def4b]::ty::context::TyCtxt>::resolver_for_lowering
  39:     0xffae641ebd7c - <rustc_middle[2c6500b4474def4b]::ty::context::GlobalCtxt>::enter::<rustc_driver_impl[233ea3aee64cd688]::run_compiler::{closure#0}::{closure#1}::{closure#3}, &rustc_data_structures[c3ef3f3d456babe4]::steal::Steal<(rustc_middle[2c6500b4474def4b]::ty::ResolverAstLowering, alloc[b23c587ff69f26b0]::sync::Arc<rustc_ast[ba5aba888c7ba019]::ast::Crate>)>>
  40:     0xffae641c3840 - <rustc_interface[45f6ee680972b70c]::interface::Compiler>::enter::<rustc_driver_impl[233ea3aee64cd688]::run_compiler::{closure#0}::{closure#1}, core[2b1f2bbd3b605c]::result::Result<core[2b1f2bbd3b605c]::option::Option<rustc_interface[45f6ee680972b70c]::queries::Linker>, rustc_span[d6938c8799c78ecd]::ErrorGuaranteed>>
  41:     0xffae64229100 - rustc_span[d6938c8799c78ecd]::create_session_globals_then::<core[2b1f2bbd3b605c]::result::Result<(), rustc_span[d6938c8799c78ecd]::ErrorGuaranteed>, rustc_interface[45f6ee680972b70c]::util::run_in_thread_with_globals<rustc_interface[45f6ee680972b70c]::util::run_in_thread_pool_with_globals<rustc_interface[45f6ee680972b70c]::interface::run_compiler<core[2b1f2bbd3b605c]::result::Result<(), rustc_span[d6938c8799c78ecd]::ErrorGuaranteed>, rustc_driver_impl[233ea3aee64cd688]::run_compiler::{closure#0}>::{closure#1}, core[2b1f2bbd3b605c]::result::Result<(), rustc_span[d6938c8799c78ecd]::ErrorGuaranteed>>::{closure#0}, core[2b1f2bbd3b605c]::result::Result<(), rustc_span[d6938c8799c78ecd]::ErrorGuaranteed>>::{closure#0}::{closure#0}::{closure#0}>
  42:     0xffae64223c04 - std[a32bae77f370052a]::sys::backtrace::__rust_begin_short_backtrace::<rustc_interface[45f6ee680972b70c]::util::run_in_thread_with_globals<rustc_interface[45f6ee680972b70c]::util::run_in_thread_pool_with_globals<rustc_interface[45f6ee680972b70c]::interface::run_compiler<core[2b1f2bbd3b605c]::result::Result<(), rustc_span[d6938c8799c78ecd]::ErrorGuaranteed>, rustc_driver_impl[233ea3aee64cd688]::run_compiler::{closure#0}>::{closure#1}, core[2b1f2bbd3b605c]::result::Result<(), rustc_span[d6938c8799c78ecd]::ErrorGuaranteed>>::{closure#0}, core[2b1f2bbd3b605c]::result::Result<(), rustc_span[d6938c8799c78ecd]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[2b1f2bbd3b605c]::result::Result<(), rustc_span[d6938c8799c78ecd]::ErrorGuaranteed>>
  43:     0xffae642264bc - <<std[a32bae77f370052a]::thread::Builder>::spawn_unchecked_<rustc_interface[45f6ee680972b70c]::util::run_in_thread_with_globals<rustc_interface[45f6ee680972b70c]::util::run_in_thread_pool_with_globals<rustc_interface[45f6ee680972b70c]::interface::run_compiler<core[2b1f2bbd3b605c]::result::Result<(), rustc_span[d6938c8799c78ecd]::ErrorGuaranteed>, rustc_driver_impl[233ea3aee64cd688]::run_compiler::{closure#0}>::{closure#1}, core[2b1f2bbd3b605c]::result::Result<(), rustc_span[d6938c8799c78ecd]::ErrorGuaranteed>>::{closure#0}, core[2b1f2bbd3b605c]::result::Result<(), rustc_span[d6938c8799c78ecd]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[2b1f2bbd3b605c]::result::Result<(), rustc_span[d6938c8799c78ecd]::ErrorGuaranteed>>::{closure#1} as core[2b1f2bbd3b605c]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  44:     0xffae6a41d94c - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h8f59f5b3c43759d1
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/boxed.rs:1972:9
  45:     0xffae6a41d94c - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h7541037a2f8924a1
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/boxed.rs:1972:9
  46:     0xffae6a41d94c - std::sys::pal::unix::thread::Thread::new::thread_start::hd1bb28d1647e61f1
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/sys/pal/unix/thread.rs:105:17
  47:     0xffae6d4d597c - <unknown>
  48:     0xffae6d53ba4c - <unknown>
  49:                0x0 - <unknown>

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.84.0 (9fc6b4312 2025-01-07) running on aarch64-unknown-linux-gnu

note: compiler flags: -C embed-bitcode=no -C debuginfo=2

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [resolver_for_lowering_raw] getting the resolver for lowering
end of query stack
error: could not compile `python-buildpack` (bin "python-buildpack" test)

Caused by:
  process didn't exit successfully: `/home/runner/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/bin/rustc --crate-name python_buildpack --edition=2021 src/main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 '--warn=clippy::pedantic' '--warn=clippy::unwrap_used' --warn=unused_crate_dependencies --warn=unsafe_code --warn=unreachable_pub '--allow=clippy::result_large_err' '--warn=clippy::panic_in_result_fn' '--allow=clippy::large_enum_variant' --test --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=7e31f0a805c98a00 -C extra-filename=-7e31f0a805c98a00 --out-dir /home/runner/work/buildpacks-python/buildpacks-python/target/debug/deps -L dependency=/home/runner/work/buildpacks-python/buildpacks-python/target/debug/deps --extern indoc=/home/runner/work/buildpacks-python/buildpacks-python/target/debug/deps/libindoc-730e18d3d79a7d4d.so --extern libcnb=/home/runner/work/buildpacks-python/buildpacks-python/target/debug/deps/liblibcnb-b2d41c17f48f3632.rlib --extern libcnb_test=/home/runner/work/buildpacks-python/buildpacks-python/target/debug/deps/liblibcnb_test-b23ea7d64a222cfc.rlib --extern libherokubuildpack=/home/runner/work/buildpacks-python/buildpacks-python/target/debug/deps/liblibherokubuildpack-ffceaeb7b1357dfa.rlib --extern serde=/home/runner/work/buildpacks-python/buildpacks-python/target/debug/deps/libserde-2f687dfd07f05e19.rlib --extern tar=/home/runner/work/buildpacks-python/buildpacks-python/target/debug/deps/libtar-72685db7610ce40c.rlib --extern ureq=/home/runner/work/buildpacks-python/buildpacks-python/target/debug/deps/libureq-91e39a6a9e4b9d2c.rlib --extern zstd=/home/runner/work/buildpacks-python/buildpacks-python/target/debug/deps/libzstd-1bd9d6aaaf25af48.rlib -L native=/home/runner/work/buildpacks-python/buildpacks-python/target/debug/build/ring-08ca213cbd59f37d/out -L native=/home/runner/work/buildpacks-python/buildpacks-python/target/debug/build/zstd-sys-93d8c14fdcde7b64/out` (exit status: 101)
Error: Process completed with exit code 101.

[theemathas]

Possibly caused by faulty github runners. See #135867

[edmorley]

@theemathas Thank you for the cross-link - that issue didn't come up when I searched (since the failure mode is different).

The strange thing with our failure, is that we've been happily running ARM on GitHub Actions for months using the custom/larger runner groups feature (where GitHub still host the runners, but you specify custom sizes or OSes etc) with no issue. This ICE occurred shortly after we switched to the new preview of the public ARM runners.

I'd presumed the OS image and machine types would be identical. However it seems there may be differences...

This was the runner/image metadata before (using a custom runner group):

2025-01-23T13:06:01.3387693Z Current runner version: '2.321.0'
2025-01-23T13:06:01.3397159Z Runner name: 'pub-hk-ubuntu-24.04-arm-medium_8f61aebadf2f'
2025-01-23T13:06:01.3398724Z Runner group name: 'Public Runners'
2025-01-23T13:06:01.3399951Z Machine name: 'runner'
2025-01-23T13:06:01.3424021Z ##[group]VM Image
2025-01-23T13:06:01.3425241Z OS: Linux (arm64)
2025-01-23T13:06:01.3426014Z Source: Marketplace
2025-01-23T13:06:01.3426783Z Publisher: arm
2025-01-23T13:06:01.3427610Z Offer: github_arm_linux_runner_2404
2025-01-23T13:06:01.3428587Z Plan: github_arm_linux_runner_plan_2404
2025-01-23T13:06:01.3429587Z Version: 1.0.10
2025-01-23T13:06:01.3430290Z ##[endgroup]

And this is the runner/image metadata now:

2025-01-23T13:06:10.1619450Z Current runner version: '2.321.0'
2025-01-23T13:06:10.1646119Z ##[group]Operating System
2025-01-23T13:06:10.1647123Z Ubuntu
2025-01-23T13:06:10.1647607Z 24.04.1
2025-01-23T13:06:10.1648103Z LTS
2025-01-23T13:06:10.1648674Z ##[endgroup]
2025-01-23T13:06:10.1649206Z ##[group]Runner Image
2025-01-23T13:06:10.1649764Z Image: ubuntu-24.04
2025-01-23T13:06:10.1650370Z Version: 0.0.10
2025-01-23T13:06:10.1651253Z Included Software: https://github.com/actions/partner-runner-images/blob/main/images/arm-ubuntu-24-image.md
2025-01-23T13:06:10.1652232Z ##[endgroup]
2025-01-23T13:06:10.1652794Z ##[group]Runner Image Provisioner
2025-01-23T13:06:10.1653349Z 2.0.417.1
2025-01-23T13:06:10.1653821Z ##[endgroup]