protoc segfaults on macOS 12 Beta if built with bazel

[f355]

Hi!

I'm using protobuf as a Bazel dependency in my project and since upgrading to macOS Monterey Beta protoc has started segfaulting.

I'm running macOS Monterey 12.0 Beta (21A5304g), x86_64, with Command Line Tools for Xcode 13 beta 5 and Bazel 4.1.0.

Steps to reproduce:

1. clone this repo
2. bazel build //:protoc
3. ./bazel-bin/protoc
4. it crashes with Segmentation fault (11) instead of printing the usage screen. Needless to say, invoking it with real arguments crashes as well.

I've tried building it with autotools and it works just fine, so I assume it's something with the compiler/linker options in Bazel that breaks things. Unfortunately, my clang-fu is not nearly strong enough to figure out the cause myself.

Here's the backtrace from lldb:

$ lldb ./bazel-bin/protoc
(lldb) target create "./bazel-bin/protoc"
Current executable set to '/Users/konstantin/src/protobuf/bazel-bin/protoc' (x86_64).
(lldb) run
Process 54067 launched: '/Users/konstantin/src/protobuf/bazel-bin/protoc' (x86_64)
Process 54067 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xffffffffffffffe8)
    frame #0: 0x00007ff80194ac3c libc++.1.dylib`std::__1::basic_ostream<char, std::__1::char_traits<char> >::sentry::sentry(std::__1::basic_ostream<char, std::__1::char_traits<char> >&) + 16
libc++.1.dylib`std::__1::basic_ostream<char, std::__1::char_traits<char> >::sentry::sentry:
->  0x7ff80194ac3c <+16>: movq   -0x18(%rax), %rax
    0x7ff80194ac40 <+20>: cmpl   $0x0, 0x20(%rsi,%rax)
    0x7ff80194ac45 <+25>: jne    0x7ff80194ac5f            ; <+51>
    0x7ff80194ac47 <+27>: movq   %rdi, %rbx
Target 0: (protoc) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xffffffffffffffe8)
  * frame #0: 0x00007ff80194ac3c libc++.1.dylib`std::__1::basic_ostream<char, std::__1::char_traits<char> >::sentry::sentry(std::__1::basic_ostream<char, std::__1::char_traits<char> >&) + 16
    frame #1: 0x00000001000201a7 protoc`std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::__put_character_sequence<char, std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, char const*, unsigned long) + 39
    frame #2: 0x0000000100006889 protoc`std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::operator<<<std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, char const*) + 57
    frame #3: 0x00000001000118fd protoc`google::protobuf::compiler::CommandLineInterface::PrintHelpText() + 45
    frame #4: 0x000000010000bc01 protoc`google::protobuf::compiler::CommandLineInterface::ParseArguments(int, char const* const*) + 497
    frame #5: 0x000000010000a119 protoc`google::protobuf::compiler::CommandLineInterface::Run(int, char const* const*) + 57
    frame #6: 0x00000001000021d9 protoc`google::protobuf::compiler::ProtobufMain(int, char**) + 1785
    frame #7: 0x0000000100002892 protoc`main + 34
    frame #8: 0x0000000100bd54d5 dyld`start + 421

[f355]

for what it's worth, it somehow works with the full XCode, but not with the Command Line Tools.

[elharo]

None of us have Mac OS beta builds to play with. I suppose it could be a bug in Monterey that should be reported ot Apple, especially given that it works with full XCode, but I really don't know. Since it can be built with the full XCode, I suspect this is low priority and we'll try to take a look when we uograde to the released Monterey if it's still happening then.

[aarone]

still happening after public Monterey release

[aarone]

after some investigation, looks like we compile protoc in bazel with a nix-provided compiler which seems to be the source of my issue with Monterey; may not be reproducible with the same steps originally reported. Sorry for any confusion.

[jvolkman]

This looks like a more general Bazel / macOS 12 issue. If Xcode 13 isn't installed (which is apparently required on Monterey), Bazel's default cc toolchain builds binaries that segfault.

[elharo]

Thanks. Perhaps report to bazel then.

[kalbasit]

after some investigation, looks like we compile protoc in bazel with a nix-provided compiler which seems to be the source of my issue with Monterey; may not be reproducible with the same steps originally reported. Sorry for any confusion.

I'm having the same issue. Nix is on 21.05, and Bazel 4.1.0.

% bazel build @com_github_protocolbuffers_protobuf//:protobuf_python
...
(11:43:13) INFO: Analyzed target @com_github_protocolbuffers_protobuf//:protobuf_python (32 packages loaded, 678 targets configured).
(11:43:13) INFO: Found 1 target...
(11:44:18) INFO: From Linking external/com_github_protocolbuffers_protobuf/libprotobuf_lite.a [for host]:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: file: bazel-out/host/bin/external/com_github_protocolbuffers_protobuf/_objs/protobuf_lite/io_win32.o has no symbols
(11:44:24) INFO: From Compiling src/google/protobuf/descriptor.cc [for host]:
external/com_github_protocolbuffers_protobuf/src/google/protobuf/descriptor.cc:1038:9: warning: passing variable 'unknown_enum_values_by_number_' by reference requires holding mutex 'unknown_enum_values_mu_' [-Wthread-safety-reference]
        unknown_enum_values_by_number_, std::make_pair(parent, number));
        ^
external/com_github_protocolbuffers_protobuf/src/google/protobuf/descriptor.cc:1048:9: warning: passing variable 'unknown_enum_values_by_number_' by reference requires holding mutex 'unknown_enum_values_mu_' [-Wthread-safety-reference]
        unknown_enum_values_by_number_, std::make_pair(parent, number));
        ^
2 warnings generated.
(11:44:24) INFO: From Linking external/com_github_protocolbuffers_protobuf/libprotobuf.a [for host]:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: file: bazel-out/host/bin/external/com_github_protocolbuffers_protobuf/_objs/protobuf/error_listener.o has no symbols
(11:44:25) ERROR: /Users/yl/.cache/ktmr/bazel/d0e47671157ea8ea9a770891e4f3ebca/external/com_github_protocolbuffers_protobuf/BUILD:898:17: ProtoCompile external/com_github_protocolbuffers_protobuf/python/google/protobuf/api_pb2.py failed: (Segmentation fault): protoc failed: error executing command
  (cd /Users/yl/.cache/ktmr/bazel/d0e47671157ea8ea9a770891e4f3ebca/sandbox/processwrapper-sandbox/196/execroot/kt && \
  exec env - \
    LD_LIBRARY_PATH=/nix/store/k4kla1x6idyl1rbjj4r3vxzywipiy0sk-native-dependencies-env/lib \
    PATH=/Users/yl/code/KeepTruckin/kt/bin:/nix/store/l73jzkvkyksf88ig00d4xmdb82r2b932-bash-interactive-5.1-p8/bin:/nix/store/1nxk828hqq7zbyr7kqs0q95z7fa9ilz2-pkg-config-wrapper-0.29.2/bin:/nix/store/5yd5ydlzajm3m8pa0frc77zpfvwczf1r-ktmr-final-environment/bin:/nix/store/9q2s3gm5hyv02r2290sdx1nxab8haiym-coreutils-8.32/bin:/nix/store/rz7l8z4k85ghf1ppv4s4gnj4az38vils-findutils-4.7.0/bin:/nix/store/06pc16i3b5nnsykkhpmr8ygv7jpsbmdm-diffutils-3.7/bin:/nix/store/jlq73mwi01bizq7rmz0iw45f47xywa5k-gnused-4.8/bin:/nix/store/5f5h7wi680wfq7qmyy2kjgba03sq59gx-gnugrep-3.6/bin:/nix/store/yayh4fd582y9rbgnr22jw01cgfcphid2-gawk-5.1.0/bin:/nix/store/9i3jamddqlgdajzh0zx5s02cjg9n07bf-gnutar-1.34/bin:/nix/store/p9bxax4hb73q39jxgnb6s3hf8kqjbprl-gzip-1.10/bin:/nix/store/r7a8rxds7hk3s1g0sbk2p8pxajnknz8h-bzip2-1.0.6.0.2-bin/bin:/nix/store/2ykgxldqvv11pdq8axjl8p9xq2iyiqp0-gnumake-4.3/bin:/nix/store/c9cxq1583a85bsq76q2rbnbiwwp7ygxr-bash-4.4-p23/bin:/nix/store/08zvbs3k26mvik3fscp3r5vwbssd6ij1-patch-2.7.6/bin:/nix/store/kmcjpbz1zmnj2j9ar4pw0092gcy2xszb-xz-5.2.5-bin/bin:/Users/yl/code/KeepTruckin/kt/bin:/Users/yl/.nix-profile/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/nix/store/c9cxq1583a85bsq76q2rbnbiwwp7ygxr-bash-4.4-p23/bin:/nix/store/9q2s3gm5hyv02r2290sdx1nxab8haiym-coreutils-8.32/bin:/nix/store/rz7l8z4k85ghf1ppv4s4gnj4az38vils-findutils-4.7.0/bin:/nix/store/yayh4fd582y9rbgnr22jw01cgfcphid2-gawk-5.1.0/bin:/nix/store/5f5h7wi680wfq7qmyy2kjgba03sq59gx-gnugrep-3.6/bin:/nix/store/xgplkrnxq8a8b1drdw1z0k7hhk5c3nwh-gnutar-1.34/bin:/nix/store/jlq73mwi01bizq7rmz0iw45f47xywa5k-gnused-4.8/bin:/nix/store/p9bxax4hb73q39jxgnb6s3hf8kqjbprl-gzip-1.10/bin:/nix/store/sl06xm48j3llacrf3nibsjfiissx1w83-which-2.21/bin:/nix/store/kznnfydbdn3f541iy888za0cq8paw9li-unzip-6.0/bin:/nix/store/sd71pdjvn60nc850q5mc44gl1kjhcp68-file-5.39/bin:/nix/store/n9y7gkgjyy1s2wv3r17q3myf3z6lhpfy-zip-3.0/bin:/nix/store/90gbhwchm06a2vx5v4qqibfrj5f67v4r-python-2.7.18/bin:/nix/store/d6x0vlghi8ma42f81981lhz516y60qk4-python3-3.8.9/bin \
  bazel-out/host/bin/external/com_github_protocolbuffers_protobuf/protoc '--python_out=bazel-out/darwin-fastbuild/bin/external/com_github_protocolbuffers_protobuf/python' -Iexternal/com_github_protocolbuffers_protobuf/python -Ibazel-out/darwin-fastbuild/bin/external/com_github_protocolbuffers_protobuf/python bazel-out/darwin-fastbuild/bin/external/com_github_protocolbuffers_protobuf/python/google/protobuf/api.proto)
...

Were you able to address this issue @aarone?

[aarone]

have you installed Xcode 13 to upgrade cc to 13.0?

[kalbasit]

I did, it didn't help though. I'm reinstalling the OS right now, I'll try again one more time.

[pcj]

I don't think this issue should be closed. This is affecting lots of people and we'll probably need google C++ experts to help.

[pcj]

ERROR: /Users/.../v1/BUILD.bazel:18:14: Generating Descriptor Set proto_library //api/v1:v1_proto failed: 

(Segmentation fault): protoc failed: error executing command bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/protoc 
'--proto_path=bazel-out/darwin-fastbuild/bin/external/com_google_protobuf/_virtual_imports/descriptor_proto' ... 

[elharo]

This appears to be an issue with bazel, not with the protobuf project. This is not the correct repo in which to address that.

[pcj]

@elharo perhaps, but there /is/ quite a bit of overlap, and I suspect the people that maintain this repo are in the best position to help get to the bottom of this.

[jvolkman]

For anyone following, this has been fixed upstream in Bazel 4.2.2.

bazelbuild/bazel#14275