Invalid metadata for arm64e due to Xcode 15+ on ARM64e macOS

[arttet]

I tried to compile arm64e-apple-darwin on aarch64 (ARM-based CPU) macOS. So, I got the following error

error[E0786]: found invalid metadata files for crate `zerocopy_derive`

  error[E0786]: found invalid metadata files for crate `zerocopy_derive`
     --> /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/zerocopy-0.7.35/src/lib.rs:285:9
      |
  285 | pub use zerocopy_derive::Unaligned;
      |         ^^^^^^^^^^^^^^^
      |
      = note: no `.rustc` section in '/Users/runner/work/rust-compiler-builder/rust-compiler-builder/rust/build/aarch64-apple-darwin/stage1-rustc/arm64e-apple-darwin/release/deps/libzerocopy_derive-85cad85f9873b5cd.dylib'
  
  error[E0786]: found invalid metadata files for crate `zerocopy_derive`
     --> /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/zerocopy-0.7.35/src/lib.rs:293:9
      |
  293 | pub use zerocopy_derive::KnownLayout;
      |         ^^^^^^^^^^^^^^^
      |
      = note: no `.rustc` section in '/Users/runner/work/rust-compiler-builder/rust-compiler-builder/rust/build/aarch64-apple-darwin/stage1-rustc/arm64e-apple-darwin/release/deps/libzerocopy_derive-85cad85f9873b5cd.dylib'
  
  error[E0786]: found invalid metadata files for crate `zerocopy_derive`
      --> /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/zerocopy-0.7.35/src/lib.rs:1118:9
       |
  1118 | pub use zerocopy_derive::FromZeroes;
       |         ^^^^^^^^^^^^^^^
       |
       = note: no `.rustc` section in '/Users/runner/work/rust-compiler-builder/rust-compiler-builder/rust/build/aarch64-apple-darwin/stage1-rustc/arm64e-apple-darwin/release/deps/libzerocopy_derive-85cad85f9873b5cd.dylib'
  
  error[E0786]: found invalid metadata files for crate `zerocopy_derive`
      --> /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/zerocopy-0.7.35/src/lib.rs:1733:9
       |
  1733 | pub use zerocopy_derive::FromBytes;
       |         ^^^^^^^^^^^^^^^
       |
       = note: no `.rustc` section in '/Users/runner/work/rust-compiler-builder/rust-compiler-builder/rust/build/aarch64-apple-darwin/stage1-rustc/arm64e-apple-darwin/release/deps/libzerocopy_derive-85cad85f9873b5cd.dylib'
  
  error[E0786]: found invalid metadata files for crate `zerocopy_derive`
      --> /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/zerocopy-0.7.35/src/lib.rs:2599:9
       |
  2599 | pub use zerocopy_derive::AsBytes;
       |         ^^^^^^^^^^^^^^^
       |
       = note: no `.rustc` section in '/Users/runner/work/rust-compiler-builder/rust-compiler-builder/rust/build/aarch64-apple-darwin/stage1-rustc/arm64e-apple-darwin/release/deps/libzerocopy_derive-85cad85f9873b5cd.dylib'
  
  error[E0432]: unresolved import `crate::KnownLayout`
    --> /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/zerocopy-0.7.35/src/util.rs:21:34
     |
  21 |     use crate::{util::AsAddress, KnownLayout, _CastType};
     |                                  ^^^^^^^^^^^
  
  [RUSTC-TIMING] displaydoc test:false 7.021
     Compiling tracing-attributes v0.1.27
  error[E0277]: the trait bound `byteorder::U16<O>: FromZeroes` is not satisfied
     --> /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/zerocopy-0.7.35/src/byteorder.rs:292:49
      |
  236 | / macro_rules! define_type {
  237 | |     ($article:ident,
  238 | |         $name:ident,
  239 | |         $native:ident,
  ...   |
  292 | |             impl_or_verify!(O => FromZeroes for $name<O>);
      | |                                                 ^^^^^^^^ the trait `FromZeroes` is not implemented for `byteorder::U16<O>`
  ...   |
  462 | |     };
  463 | | }
      | |_- in this expansion of `define_type!`
  464 |
  465 | / define_type!(
  466 | |     A,
  467 | |     U16,
  468 | |     u16,
  ...   |
  477 | |     [U32, U64, U128]
  478 | | );
      | |_- in this macro invocation
      |
      = help: the following other types implement trait `FromZeroes`:
                ()
                *const T
                *mut T
                ManuallyDrop<T>
                MaybeUninit<T>
                Option<&T>
                Option<&mut T>
                Option<NonNull<T>>
              and 103 others
  note: required by a bound in `byteorder::_::_::Subtrait`
     --> /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/zerocopy-0.7.35/src/macros.rs:323:29
      |
  28  | / macro_rules! safety_comment {
  29  | |     (#[doc = r" SAFETY:"] $($(#[$attr:meta])* $macro:ident!$args:tt;)*) => {
  30  | |         #[allow(clippy::undocumented_unsafe_blocks, unused_attributes)]
  31  | |         const _: () = { $($(#[$attr])* $macro!$args;)* };
      | |                                        ------------ in this macro invocation (#3)
  32  | |     }
  33  | | }
      | |_- in this expansion of `safety_comment!` (#2)
  ...
  279 | / macro_rules! impl_or_verify {
  280 | |     // The following two match arms follow the same pattern as their
  281 | |     // counterparts in `unsafe_impl!`; see the documentation on those arms for
  282 | |     // more details.
  ...   |
  302 | /         impl_or_verify!(@verify $trait, {
  303 |               impl<$($tyvar $(: $(? $optbound +)* $($bound +)*)?),*> Subtrait for $ty {}
  304 | |         });
      | |__________- in this macro invocation (#4)
  ...
  323 | |             trait Subtrait: $trait {}
      | |                             ^^^^^^ required by this bound in `Subtrait`
  ...   |
  326 | |     };
  327 | | }
      | | -
      | |_|
      | |_in this expansion of `impl_or_verify!` (#3)
      |   in this expansion of `impl_or_verify!` (#4)
      |
     ::: /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/zerocopy-0.7.35/src/byteorder.rs:236:1
      |
  236 | / macro_rules! define_type {
  237 | |     ($article:ident,
  238 | |         $name:ident,
  239 | |         $native:ident,
  ...   |
  287 | /         safety_comment! {
  288 |               /// SAFETY:
  289 |               /// `$name<O>` is `repr(transparent)`, and so it has the same layout
  290 |               /// as its only non-zero field, which is a `u8` array. `u8` arrays
  ...
  295 |               impl_or_verify!(O => Unaligned for $name<O>);
  296 | |         }
      | |_________- in this macro invocation (#2)
  ...
  462 | |     };
  463 | | }
      | |_- in this expansion of `define_type!` (#1)
  464 |
  465 | / define_type!(
  466 | |     A,
  467 | |     U16,
  468 | |     u16,
  ...   |
  477 | |     [U32, U64, U128]
  478 | | );
      | |_- in this macro invocation (#1)
  
  error[E0277]: the trait bound `byteorder::U16<O>: FromBytes` is not satisfied
     --> /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/zerocopy-0.7.35/src/byteorder.rs:293:48
      |
  236 | / macro_rules! define_type {
  237 | |     ($article:ident,
  238 | |         $name:ident,
  239 | |         $native:ident,
  ...   |
  293 | |             impl_or_verify!(O => FromBytes for $name<O>);
      | |                                                ^^^^^^^^ the trait `FromBytes` is not implemented for `byteorder::U16<O>`
  ...   |
  462 | |     };
  463 | | }
      | |_- in this expansion of `define_type!`
  464 |
  465 | / define_type!(
  466 | |     A,
  467 | |     U16,
  468 | |     u16,
  ...   |
  477 | |     [U32, U64, U128]
  478 | | );
      | |_- in this macro invocation
      |
      = help: the following other types implement trait `FromBytes`:
                ()
                ManuallyDrop<T>
                MaybeUninit<T>
                Option<NonZero<i128>>
                Option<NonZero<i16>>
                Option<NonZero<i32>>
                Option<NonZero<i64>>
                Option<NonZero<i8>>
              and 69 others
  note: required by a bound in `byteorder::_::_::Subtrait`
     --> /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/zerocopy-0.7.35/src/macros.rs:323:29
      |
  28  | / macro_rules! safety_comment {
  29  | |     (#[doc = r" SAFETY:"] $($(#[$attr:meta])* $macro:ident!$args:tt;)*) => {
  30  | |         #[allow(clippy::undocumented_unsafe_blocks, unused_attributes)]
  31  | |         const _: () = { $($(#[$attr])* $macro!$args;)* };
      | |                                        ------------ in this macro invocation (#3)
  32  | |     }
  33  | | }
      | |_- in this expansion of `safety_comment!` (#2)
  ...
  279 | / macro_rules! impl_or_verify {
  280 | |     // The following two match arms follow the same pattern as their
  281 | |     // counterparts in `unsafe_impl!`; see the documentation on those arms for
  282 | |     // more details.
  ...   |
  302 | /         impl_or_verify!(@verify $trait, {
  303 |               impl<$($tyvar $(: $(? $optbound +)* $($bound +)*)?),*> Subtrait for $ty {}
  304 | |         });
      | |__________- in this macro invocation (#4)
  ...
  323 | |             trait Subtrait: $trait {}
      | |                             ^^^^^^ required by this bound in `Subtrait`
  ...   |
  326 | |     };
  327 | | }
      | | -
      | |_|
      | |_in this expansion of `impl_or_verify!` (#3)
      |   in this expansion of `impl_or_verify!` (#4)
      |
     ::: /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/zerocopy-0.7.35/src/byteorder.rs:236:1
      |
  236 | / macro_rules! define_type {
  237 | |     ($article:ident,
  238 | |         $name:ident,
  239 | |         $native:ident,
  ...   |
  287 | /         safety_comment! {
  288 |               /// SAFETY:
  289 |               /// `$name<O>` is `repr(transparent)`, and so it has the same layout
  290 |               /// as its only non-zero field, which is a `u8` array. `u8` arrays
  ...
  295 |               impl_or_verify!(O => Unaligned for $name<O>);
  296 | |         }
      | |_________- in this macro invocation (#2)
  ...
  462 | |     };
  463 | | }
      | |_- in this expansion of `define_type!` (#1)
  464 |
  465 | / define_type!(
  466 | |     A,
  467 | |     U16,
  468 | |     u16,
  ...   |
  477 | |     [U32, U64, U128]
  478 | | );
      | |_- in this macro invocation (#1)
  
  error[E0277]: the trait bound `byteorder::U16<O>: AsBytes` is not satisfied
     --> /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/zerocopy-0.7.35/src/byteorder.rs:294:46
      |
  236 | / macro_rules! define_type {
  237 | |     ($article:ident,
  238 | |         $name:ident,
  239 | |         $native:ident,
  ...   |
  294 | |             impl_or_verify!(O => AsBytes for $name<O>);
      | |                                              ^^^^^^^^ the trait `AsBytes` is not implemented for `byteorder::U16<O>`
  ...   |
  462 | |     };
  463 | | }
      | |_- in this expansion of `define_type!`
  464 |
  465 | / define_type!(
  466 | |     A,
  467 | |     U16,
  468 | |     u16,
  ...   |
  477 | |     [U32, U64, U128]
  478 | | );
      | |_- in this macro invocation
      |

This issue can be reproduced only on aarch64 (ARM-based CPU) macOS.

See google/zerocopy#1739
See llvm/llvm-project#79543

Blocked by llvm/llvm-project#80200
Relates to #130085

[arttet]

Hi @bjorn3,

Could you please share your thoughts about this issue?

[bjorn3]

It seems that Apple has changed the Mach-O machine identifier for arm64e with the latest XCode version: #130085 Rustc still uses the old machine identifier for the metadata object file, which would cause the linker to ignore it with the latest XCode version. What is surprising to me however if you are using the latest XCode version is that linking would succeed at all as LLVM also uses the old machine identifier.

[arttet]

Thank you so much! It seems you are right. I forgot about it because we didn't get linker errors.

What is surprising to me however if you are using the latest XCode version is that linking would succeed at all as LLVM also uses the old machine identifier.

I have two builds. One of them has use-lld = true, another has use-lld = false.

config.toml

# Indicates whether LLD will be compiled and made available in the sysroot for rustc to execute, and
# whether to set it as rustc's default linker on `x86_64-unknown-linux-gnu`. This will also only be
# when *not* building an external LLVM (so only when using `download-ci-llvm` or building LLVM from
# the in-tree source): setting `llvm-config` in the `[target.x86_64-unknown-linux-gnu]` section will
# make this default to false.
lld = true

# Indicates whether LLD will be used to link Rust crates during bootstrap on
# supported platforms.
# If set to `true` or `"external"`, a global `lld` binary that has to be in $PATH
# will be used.
# If set to `"self-contained"`, rust-lld from the snapshot compiler will be used.
#
# On MSVC, LLD will not be used if we're cross linking.
#
# Explicitly setting the linker for a target will override this option when targeting MSVC.
use-lld = false

lld is enabled by default. But use-lld is not enabled by default.
So, lld is enough on x86_64-apple-darwin because we build only library artifacts.

Building stage2 library artifacts (x86_64-apple-darwin -> arm64e-apple-darwin)
[TIMING] core::build_steps::compile::Std { target: arm64e-apple-darwin, compiler: Compiler { stage: 2, host: x86_64-apple-darwin }, crates: [], force_recompile: false, extra_rust_args: [], is_for_mir_opt_tests: false } -- 0.362

But if we build the target with this configuration on arm64e-apple-darwin, we will have these side effects.

Building compiler artifacts (stage1 -> stage2, aarch64-apple-darwin)
error[E0786]: found invalid metadata files for crate `zerocopy_derive`

So, we have to use use-lld = true on arm64e-apple-darwin.

Would it be a good idea to add some checks to bootstrap for these targets?

[apiraino]

WG-prioritization assigning priority (Zulip discussion).

cc @rust-lang/bootstrap about the suggestion to forcing using use-lld = true on arm64e-apple-darwin

@rustbot label -I-prioritize +P-high