Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bazel 4.2.2: C fails to link on macOS x86_64 #14395

Closed
uhthomas opened this issue Dec 8, 2021 · 8 comments
Closed

Bazel 4.2.2: C fails to link on macOS x86_64 #14395

uhthomas opened this issue Dec 8, 2021 · 8 comments
Assignees

Comments

@uhthomas
Copy link
Contributor

uhthomas commented Dec 8, 2021

Description of the problem / feature request:

C fails to link on macOS x86_64. This works fine on Bazel 4.2.1 and so I would consider this a regression.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I have a repository which contains a reproduction for the issue.

https://github.com/uhthomas/bazel-issue-14395

git clone https://github.com/uhthomas/bazel-issue-14395
cd bazel-issue-14395
USE_BAZEL_VERSION=4.2.2 bazel build //:bin

The project builds okay with Bazel 4.2.1.

USE_BAZEL_VERSION=4.2.1 bazel build //:bin

What operating system are you running Bazel on?

❯ uname -mrs
Darwin 20.6.0 x86_64

What's the output of bazel info release?

❯ USE_BAZEL_VERSION=4.2.2 bazel info release
release 4.2.2

Have you found anything relevant by searching the web?

No.

Any other information, logs, or outputs that you want to share?

❯ USE_BAZEL_VERSION=4.2.2 bazel build //:bin
INFO: Analyzed target //:bin (46 packages loaded, 8076 targets configured).
INFO: Found 1 target...
ERROR: /private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/raze__memchr__2_4_1/BUILD.bazel:40:19: Compiling Rust bin memchr_build_script_ v2.4.1 (35 files) failed: (Exit 1): process_wrapper failed: error executing command bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/rules_rust/util/process_wrapper/process_wrapper --subst 'pwd=${pwd}' -- external/rust_darwin_x86_64/bin/rustc external/raze__memchr__2_4_1/build.rs ... (remaining 23 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
error: linking with `external/local_config_cc/cc_wrapper.sh` failed: exit status: 1
  |
  = note: "external/local_config_cc/cc_wrapper.sh" "-m64" "-arch" "x86_64" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.memchr_build_script_.ae7338a9-cgu.0.rcgu.o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.memchr_build_script_.ae7338a9-cgu.1.rcgu.o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.memchr_build_script_.ae7338a9-cgu.10.rcgu.o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.memchr_build_script_.ae7338a9-cgu.2.rcgu.o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.memchr_build_script_.ae7338a9-cgu.3.rcgu.o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.memchr_build_script_.ae7338a9-cgu.4.rcgu.o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.memchr_build_script_.ae7338a9-cgu.5.rcgu.o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.memchr_build_script_.ae7338a9-cgu.6.rcgu.o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.memchr_build_script_.ae7338a9-cgu.7.rcgu.o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.memchr_build_script_.ae7338a9-cgu.8.rcgu.o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.memchr_build_script_.ae7338a9-cgu.9.rcgu.o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.1s71bpdk8li7d4tg.rcgu.o" "-L" "external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib" "-L" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libstd-dd8a82589e0cba34.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libpanic_unwind-8c04c8bd0d1a8900.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libobject-c6a4ae86ed2c40d0.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libmemchr-f9ab4d1b2e38b05e.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libaddr2line-002c7b677ad6c512.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libgimli-a3f3d9f86c37973f.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libstd_detect-8b14bcf2354140fd.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/librustc_demangle-d6f2fd91ec8bbbcc.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libhashbrown-24c80e37fb5b15c5.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_alloc-edb9b11fa36b4795.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libunwind-769780536fb7ef9b.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libcfg_if-d37c37a3a3ac2b0c.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/liblibc-c1bdc4c1f89760ef.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/liballoc-750380e9c94de9ce.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_core-1108e622f5a15c3d.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libcore-43af7053e70b1eed.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libcompiler_builtins-3a81ebf6a3abbdee.rlib" "-lSystem" "-lresolv" "-lc" "-lm" "-liconv" "-L" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib" "-o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_" "-Wl,-dead_strip" "-nodefaultlibs" "-undefined" "dynamic_lookup" "-headerpad_max_install_names" "-lstdc++" "-lm"
  = note: ld: library not found for -lstdc++
          clang: error: linker command failed with exit code 1 (use -v to see invocation)


error: aborting due to previous error

Target //:bin failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/raze__memchr__2_4_1/BUILD.bazel:65:13 Compiling Rust rlib memchr v2.4.1 (35 files) failed: (Exit 1): process_wrapper failed: error executing command bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/rules_rust/util/process_wrapper/process_wrapper --subst 'pwd=${pwd}' -- external/rust_darwin_x86_64/bin/rustc external/raze__memchr__2_4_1/build.rs ... (remaining 23 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
INFO: Elapsed time: 12.594s, Critical Path: 2.18s
INFO: 46 processes: 40 internal, 6 darwin-sandbox.
FAILED: Build did NOT complete successfully
@gabrielrussoc
Copy link

gabrielrussoc commented Dec 9, 2021

maybe bazelbuild/rules_rust#978?

4.2.2 includes #14319, which says "This will cause the values passed in the link_libs parameters to not get silently dropped on macOS 12 Monterey" - so I wonder if this just surfaced a problem in rules_rust

@uhthomas
Copy link
Contributor Author

uhthomas commented Dec 9, 2021

Thanks for taking a look! I had assumed #14319 may have been related, but not sure how or in what capacity.

"This will cause the values passed in the link_libs parameters to not get silently dropped on macOS 12 Monterey"

I'm on Big Sur.

@aiuto
Copy link
Contributor

aiuto commented Dec 15, 2021

Do you have a reproduction that does not involve rust?

@Silic0nS0ldier
Copy link
Contributor

Silic0nS0ldier commented Jan 8, 2022

Similar problem has been reported elsewhere.

Consensus seems to be that -lc++ is what should be used on MacOS.
EDIT: libstdc++ deprecated in XCode 8, removed in XCode 10 (see XCode 10 release notes).

I'm not sure where the flags come from, but I know they are put into the failing command at:
https://github.com/bazelbuild/rules_rust/blob/412826645587d2c5ea8e30a98522c25a8a53ae0c/rust/private/rustc.bzl#L648

As an experiment I tried replacing the arg with the apparent new version, and the issue was resolved.

fixed_link_args = [x for x in link_args if x != "-lstdc++"]
fixed_link_args.append("-lc++")
rustc_flags.add_joined("--codegen", fixed_link_args, join_with = " ", format_joined = "link-args=%s")

Removing the line altogether also worked (simple program though, so lib may just not have been needed).

@Silic0nS0ldier
Copy link
Contributor

Did some more digging and found where -lstdc++ was coming from. An external repository generated by cc_autoconf, called local_config_cc in my case.

Rule definition is at

cc_autoconf = repository_rule(
environ = [
"ABI_LIBC_VERSION",
"ABI_VERSION",
"BAZEL_COMPILER",
"BAZEL_HOST_SYSTEM",
"BAZEL_CXXOPTS",
"BAZEL_LINKOPTS",
"BAZEL_LINKLIBS",
"BAZEL_LLVM_COV",
"BAZEL_PYTHON",
"BAZEL_SH",
"BAZEL_TARGET_CPU",
"BAZEL_TARGET_LIBC",
"BAZEL_TARGET_SYSTEM",
"BAZEL_USE_CPP_ONLY_TOOLCHAIN",
"BAZEL_USE_XCODE_TOOLCHAIN",
"BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN",
"BAZEL_USE_LLVM_NATIVE_COVERAGE",
"BAZEL_LLVM",
"BAZEL_IGNORE_SYSTEM_HEADERS_VERSIONS",
"USE_CLANG_CL",
"CC",
"CC_CONFIGURE_DEBUG",
"CC_TOOLCHAIN_NAME",
"CPLUS_INCLUDE_PATH",
"DEVELOPER_DIR",
"GCOV",
"HOMEBREW_RUBY_PATH",
"SYSTEMROOT",
"USER",
] + MSVC_ENVVARS,
implementation = cc_autoconf_impl,
configure = True,
)

With the more limited search range I was able to identify 2 likely places -lstdc++ was coming from.

bazel_linkopts = "-lstdc++:-lm"

Argument ordering has me thinking bazel/tools/cpp/cc_toolchain_config.bzl is the ultimate source (especially since configure_osx_toolchain calls out to it). If true, then darwin may need to be handled differently.

@keith
Copy link
Member

keith commented Jan 11, 2022

Note for macOS users with Xcode installed, you have to set BAZEL_USE_CPP_ONLY_TOOLCHAIN=1 in the environment to trigger this repro case.

I looked into this a bit in clang. So what seems to be the issue is that -lstdc++ is just invalid on darwin, but clang knows how to remap it to -lc++ for the linker. For example:

% clang -lstdc++ -###
Apple clang version 13.0.0 (clang-1300.0.29.3)
Target: x86_64-apple-darwin21.2.0
 "/Applications/Xcode-13.0.0-RC1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" "-demangle" "-lto_library" "/Applications/Xcode-13.0.0-RC1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib" "-dynamic" "-arch" "x86_64" "-platform_version" "macos" "12.0.0" "12.1" "-syslibroot" "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" "-o" "a.out" "-L/usr/local/lib" "-lc++" "-lSystem" "/Applications/Xcode-13.0.0-RC1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.0.0/lib/darwin/libclang_rt.osx.a"

You can see the arg passed to the linker here is different from what I passed. The issue in this case -nodefaultlibs is also passed, which changes this behavior and doesn't remap that argument (trace the logic for this starting here):

% clang -lstdc++ -### -nodefaultlibs
Apple clang version 13.0.0 (clang-1300.0.29.3)
InstalledDir: /Applications/Xcode-13.0.0-RC1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 "/Applications/Xcode-13.0.0-RC1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" "-demangle" "-lto_library" "/Applications/Xcode-13.0.0-RC1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib" "-dynamic" "-arch" "x86_64" "-platform_version" "macos" "12.0.0" "12.1" "-syslibroot" "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" "-o" "a.out" "-L/usr/local/lib" "-lstdc++"

Based on this comment https://github.com/llvm/llvm-project/blob/11a46b174923a2509ac620bc8ff621ecdf6135b6/clang/lib/Driver/ToolChains/Darwin.cpp#L872-L876 this has been invalid for a long time.

It looks like -nodefaultlibs is coming from rust itself https://github.com/rust-lang/rust/blob/3d57c61a9e04dcd3df633f41142009d6dcad4399/compiler/rustc_codegen_ssa/src/back/linker.rs#L635-L637 and raze is pulling the flags it passes there. This in itself seems fine, but explains why this bug surfaces with rust builds specifically.

Here's a patch to fix this that I think is safe: #14542

Can folks try the bazel built with my branch to verify? It does fix the repro case above.

keith added a commit to keith/bazel that referenced this issue Jan 11, 2022
This flag has been invalid seemingly since macOS 10.9 from 2013. By
default clang remaps this back to -lc++, but if your linkopts pass
-nodefaultlibs this is not the case, in which case this invalid flag
fails the link. This happened for rust builds where the rust driver
passes this flag intentionally.

For more info see bazelbuild#14395 (comment)
@Silic0nS0ldier
Copy link
Contributor

With BAZEL_USE_CPP_ONLY_TOOLCHAIN=1 in place to bypass my newly minted XCode install (to compile Bazel from your PR branch), I was able to confirm this fixes the issue for both @uhthomas's reproduction repo and my instance of the issue.

Thanks @keith!

bazel-io pushed a commit that referenced this issue Jan 27, 2022
This flag has been invalid seemingly since macOS 10.9 from 2013. By
default clang remaps this back to -lc++, but if your linkopts pass
-nodefaultlibs this is not the case, in which case this invalid flag
fails the link. This happened for rust builds where the rust driver
passes this flag intentionally.

For more info see #14395 (comment)

Fixes google/cargo-raze#247 #14395 bazelbuild/rules_rust#226

Closes #14542.

PiperOrigin-RevId: 424588734
brentleyjones pushed a commit to brentleyjones/bazel that referenced this issue Feb 8, 2022
This flag has been invalid seemingly since macOS 10.9 from 2013. By
default clang remaps this back to -lc++, but if your linkopts pass
-nodefaultlibs this is not the case, in which case this invalid flag
fails the link. This happened for rust builds where the rust driver
passes this flag intentionally.

For more info see bazelbuild#14395 (comment)

Fixes google/cargo-raze#247 bazelbuild#14395 bazelbuild/rules_rust#226

Closes bazelbuild#14542.

PiperOrigin-RevId: 424588734
(cherry picked from commit 8b60c90)
Wyverald pushed a commit that referenced this issue Feb 9, 2022
This flag has been invalid seemingly since macOS 10.9 from 2013. By
default clang remaps this back to -lc++, but if your linkopts pass
-nodefaultlibs this is not the case, in which case this invalid flag
fails the link. This happened for rust builds where the rust driver
passes this flag intentionally.

For more info see #14395 (comment)

Fixes google/cargo-raze#247 #14395 bazelbuild/rules_rust#226

Closes #14542.

PiperOrigin-RevId: 424588734
(cherry picked from commit 8b60c90)

Co-authored-by: Keith Smiley <keithbsmiley@gmail.com>
@keith
Copy link
Member

keith commented Apr 2, 2022

We can close this one now as the fixes have been merged into bazel 5.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants