strip binary is PATH dependent which breaks builds in MacOS

[n8henrie]

I use MacOS frequently, and I've been bitten multiple times by issues related to the fact that one must have /usr/bin/strip first on PATH, prior to any strip command that may be installed by homebrew, nix, or a custom command.

• nightly 1.78 build error on macOS for several crates #122641
• 1.77 produced release binaries die w/ SIGKILL on macos 14.4 #122902
• Setting strip = true in Cargo.toml makes build scripts fail in release mode on MacOS #110536

As I also use Linux quite a bit, I prefer having the GNU version of utilities like grep, sed, and find, and I keep these first in my PATH so that my scripts keep the behavior I'm accustomed to. However, this leads to the wrong strip also being first on PATH, which breaks things.

Rust's current behavior with strip leaves me in a bit of a tough spot. In particular with the recent change to strip by default for release builds, I've found that cargo install foo unexpectedly crashes and burns if my PATH doesn't lead with /usr/bin:....

It seems like this behavior might be coming from

rust/compiler/rustc_codegen_ssa/src/back/link.rs

Line 1102 in 47ecded

fn strip_symbols_with_external_utility<'a>(

If so, it seems like it would be trivial to prefix PATH with /usr/bin (on MacOS specifically) in this function to work around this difficult behavior in a way that shouldn't affect other OSes and that should be universally correct on MacOS.

Would such a change be reasonable to consider? If so, I'd happily work on a PR.

[workingjubilee]

@n8henrie Hello! Has a bug been filed with Homebrew about this? It would be most appropriate for the Homebrew package manager to not distribute a flawed version of binutils rather than us working around them distributing a flawed version of binutils, and this is surely causing problems for other systems programming languages (or program build systems) on macOS that try to make use of the available utilities instead of shipping their own version.

My understanding is that such a bug would be filed at https://github.com/Homebrew/homebrew-core/ right?

Likewise with Nix, I believe one would file this bug with nixpkgs, right? That would be over here: https://github.com/NixOS/nixpkgs

Working around the fact that lots of people on macOS are going to hit this problem is still on the table, though I would (personally, slightly) prefer a solution that doesn't involve overriding an explicit command for "where should we first look for binaries" (i.e. $PATH). I realize that often it is manipulated programmatically but it seems equally likely to cause problems if we simply ignore it by prefixing a different path each time.

[workingjubilee]

Also if I'm misunderstanding what you mean by "wrong strip" then please correct me! It just sounds like "wrong" here means "one that is going to invent some other behavior, or perform it inadequately, rather than correctly performing the requested debuginfo stripping on macOS".

[n8henrie]

Hi, thanks for the quick response!

I'm pretty sure that the strip binaries provided by nix and homebrew work fine when used as a standalone tool, this is a rust / cargo specific bug -- I wouldn't be surprised if it has something to do with code signing (like many, many bugs on MacOS).

I'll do a little more testing to make sure I'm not misremembering and come back with more info.

[n8henrie]

Also, there is still the issue of "a user happens to have a custom utility named strip on their path" -- it would be nice if this didn't break cargo in unexpected (and potentially difficult to track down) ways. But I suppose that would affect more than just MacOS.

[workingjubilee]

🤔 I see! I am curious how it is "supposed" to be used then...? I am not aware of our usage of strip being anything interesting.

[n8henrie]

Demonstrating the issue:

$ # nix's strip is first on my PATH
$ type -a strip
strip is /etc/profiles/per-user/n8henrie/bin/strip
strip is /run/current-system/sw/bin/strip
strip is /usr/bin/strip
$ 
$ sw_vers
ProductName:            macOS
ProductVersion:         14.3.1
BuildVersion:           23D60
$ arch
arm64
$
$ cargo init --name foo
     Created binary (application) package
$
$ # stable seems to work fine
$ cargo run
   Compiling foo v0.1.0 (/private/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.4mhKqwFaTB)
    Finished dev [unoptimized + debuginfo] target(s) in 0.33s
     Running `target/debug/foo`
Hello, world!
$ cargo run --release
   Compiling foo v0.1.0 (/private/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.4mhKqwFaTB)
    Finished release [optimized] target(s) in 0.12s
     Running `target/release/foo`
Hello, world!
$ cargo --version
cargo 1.76.0 (c84b36747 2024-01-18)
$
$ # only nightly with --release shows the issue
$ cargo +nightly --version
cargo 1.78.0-nightly (2fe739fcf 2024-03-15)
$ cargo +nightly run
   Compiling foo v0.1.0 (/private/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.4mhKqwFaTB)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.45s
     Running `target/debug/foo`
Hello, world!
$ cargo +nightly run --release
   Compiling foo v0.1.0 (/private/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.4mhKqwFaTB)
    Finished `release` profile [optimized] target(s) in 0.14s
     Running `target/release/foo`
Killed: 9
$ cargo clean
     Removed 64 files, 2.5MiB total
$ # putting /usr/bin/strip ahead of the rest of the PATH seems to fix the issue
$ PATH=$(getconf PATH):~/.cargo/bin cargo +nightly run --release
   Compiling foo v0.1.0 (/private/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.4mhKqwFaTB)
    Finished `release` profile [optimized] target(s) in 0.55s
     Running `target/release/foo`
Hello, world!

nix's strip seems to work fine by itself (compare the size pre / post):

$ cargo build
   Compiling foo v0.1.0 (/private/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.4mhKqwFaTB)
    Finished dev [unoptimized + debuginfo] target(s) in 0.35s
$ stat target/debug/foo
  File: target/debug/foo
  Size: 403592          Blocks: 792        IO Block: 4096   regular file
Device: 1,25    Inode: 618156267   Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (  501/n8henrie)   Gid: (   20/   staff)
Access: 2024-03-26 20:42:52.455995870 -0600
Modify: 2024-03-26 20:42:52.456156287 -0600
Change: 2024-03-26 20:42:52.463580716 -0600
 Birth: 2024-03-26 20:42:52.455995870 -0600
$ /etc/profiles/per-user/n8henrie/bin/strip target/debug/foo
$ echo $?
0
$ stat target/debug/foo
  File: target/debug/foo
  Size: 274800          Blocks: 640        IO Block: 4096   regular file
Device: 1,25    Inode: 618156267   Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (  501/n8henrie)   Gid: (   20/   staff)
Access: 2024-03-26 20:43:16.342834724 -0600
Modify: 2024-03-26 20:43:16.084531123 -0600
Change: 2024-03-26 20:43:16.084551332 -0600
 Birth: 2024-03-26 20:42:52.455995870 -0600

[n8henrie]

The same with the MacOS strip, for comparison:

$ stat target/debug/foo
  File: target/debug/foo
  Size: 403592          Blocks: 792        IO Block: 4096   regular file
Device: 1,25    Inode: 618157322   Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (  501/n8henrie)   Gid: (   20/   staff)
Access: 2024-03-26 20:45:37.877717578 -0600
Modify: 2024-03-26 20:45:37.877877328 -0600
Change: 2024-03-26 20:45:37.885154127 -0600
 Birth: 2024-03-26 20:45:37.877717578 -0600
$ /usr/bin/strip target/debug/foo
$ stat target/debug/foo
  File: target/debug/foo
  Size: 313904          Blocks: 616        IO Block: 4096   regular file
Device: 1,25    Inode: 618157396   Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (  501/n8henrie)   Gid: (   20/   staff)
Access: 2024-03-26 20:45:55.569270038 -0600
Modify: 2024-03-26 20:45:55.569539246 -0600
Change: 2024-03-26 20:45:55.569747038 -0600
 Birth: 2024-03-26 20:45:55.569270038 -0600

[workingjubilee]

@n8henrie Er, sorry, I'm a bit confused by your demo. Did you verify the post-stripped binaries, using the variant binutils as a standalone tool, actually run? The problem seems to be when you use cargo run and not cargo build, after all.

[workingjubilee]

The Nix binutils distribution for macOS seems to be aware of, and deliberately tries to compensate for, the strip problem: https://github.com/NixOS/nixpkgs/blob/b07adff97cbe1f994715b987385ea94b20edbf22/pkgs/os-specific/darwin/binutils/default.nix#L16-L38

I wonder why it isn't working...?

[n8henrie]

Er, sorry, I'm a bit confused by your demo.

Sorry! Let me know how I can clarify. This has bitten me 3 or so different times over the last 5 years, so I've also accumulated a few preconceived notions about what's going on, I may be jumping around or making unwarranted assumptions.

The problem seems to be when you use cargo run and not cargo build, after all.

Yes, this "hello world" build fine with cargo +nightly build --release, but the issue often crops up at build-time if there are build scripts (build.rs) involved (which obviously need to "run" at build-time):

$ cat build.rs
fn main() {
    println!("cargo:rerun-if-changed=src/main.rs");
}
$
$ cargo +nightly build --release
   Compiling foo v0.1.0 (/private/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.Vi6T1JZHWL)
error: failed to run custom build command for `foo v0.1.0 (/private/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.Vi6T1JZHWL)`

Caused by:
  process didn't exit successfully: `/private/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.Vi6T1JZHWL/target/release/build/foo-f1d47ccc88c86fe0/build-script-build` (signal: 9, SIGKILL: kill)

Did you verify the post-stripped binaries, using the variant binutils as a standalone tool, actually run?

Somehow that hadn't occurred to me (SMH) and you're right:

$ cargo build
   Compiling foo v0.1.0 (/private/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.Vi6T1JZHWL)
    Finished dev [unoptimized + debuginfo] target(s) in 0.25s
$ /etc/profiles/per-user/n8henrie/bin/strip target/debug/foo
$ target/debug/foo
Killed: 9

I get the same behavior running it on a "hello world" in C.

Looks like I need to do some more digging! Will close for now, thanks for your time!

[workingjubilee]

I think some confusion is inevitable here!

If you can produce a .nix that replicates the configuration that rearranges your PATH and produces the bad binaries, that would be lovely, but you're by no means obligated to.

[n8henrie]

I can work on a repro -- do you have access to a darwin machine?

Here's a thread I made on the same issue last year: https://users.rust-lang.org/t/nix-installed-strip-is-crashing-cargo-install/97019