CPP gdb lldb no debugging symbols found

[unship]

Please provide the following information. The more we know about your system and use case, the more easily and likely we can help.

Description of the problem / feature request / question:

i build CPP with bazel on my mac, when i debug it

bazel build -c dbg :main
gdb ./bazel-bin/main

it came out:

/main.pic.o': can't open to read symbols: No such file or directory.
(no debugging symbols found)...done.

If possible, provide a minimal example to reproduce the problem:

#WORKSPACE empty
#BUILD 
cc_binary(
 name='main',
srcs=['src/main.cc']
)
#src/main.cc
#include<iostream>
int main(){
  std::cout<<"test";
}
bazel build -c dbg :main
gdb ./bazel-bin/main

Environment info

• Operating System:
  mac 10.12.3

• Bazel version (output of bazel info release):
  release 0.4.0-homebrew

• If bazel info release returns "development version" or "(@non-git)", please tell us what source tree you compiled Bazel from; git commit hash is appreciated (git rev-parse HEAD):

Have you found anything relevant by searching the web? (e.g. GitHub issues, email threads in the bazel-discuss@googlegroups.com archive)

no

Anything else, information or logs or outputs that would be helpful?

(If they are large, please upload as attachment or provide link).

FYI:
on ubuntu, gdb work is fine, but lldb still can't find debugging symbols

[unship]

dSYM is generated,but is is empty, even when add --strip never

/Users/liyanan/dev/clion/bazel-bin/main empty dSYM file detected, dSYM was created with an executable with no debug info

[david-german-tri]

RobotLocomotion/drake developers who use OS X are experiencing this also. We are actively investigating workarounds. If we find something, I'll update this thread, but any guidance from the Bazel team would be hugely appreciated!

[david-german-tri]

Related: #327.

I'm having some preliminary success running dsymutil from osx_cc_wrapper.sh. With that technique, I can at least produce a usable .dSYM somewhere deep in the sandbox, then manually symbolicate.

[david-german-tri]

Here's an outline of a workaround we've prototyped locally:

1. For every cc_{library,binary,test}, add a genrule like the following:

genrule(
    name = "foo_dsym",
    srcs = [":foo"],
    outs = ["foo.dSYM"],
    output_to_bindir = True,
    cmd = "dsymutil $(location :foo) -o $@"
)

We have binary/library/test wrapper macros that simplify insertion of this genrule across the board.

2. Use --spawn_strategy=standalone to disable sandboxing, so that the DWARF-laden .o files will be visible to dsymutil. (Also, of course, use -c dbg, which in turn sets the compiler flag -g.)

3. If you need to support platforms other than OS X, use a select statement on the cmd, and simply touch $@ on non-Apple platforms.

This has a bunch of deficiencies. It disables sandboxing, it declares a directory output from a genrule (which is unsound), it clutters the build with extra targets, and it doesn't actually build the debug symbols on bazel build :foo. But it at least provides a way to get debug symbols on OS X. If someone else has a better way, I'd be very happy!

[sistr22]

Is the workaround above still the one we should use ?

[hlopko]

CC @c-parsons @calpeyser

[yageek]

@david-german-tri Would it be possible to give some more explanation on how to use the workaround ? I added a genrule after a cc_library library statement but I still get an empty .dSYM error. Should we also add --strip=never to the build/test command ?

[dattanchu]

I use the workaround and can generate the dSYM required to step through the executable with lldb. When we run in sandbox mode, is it possible to access the object files required by dsymutil?

[baskus]

What is the status of this? When will we get debug symbols without hacks?

Thanks! :)

[jmarantz]

+1 -- I had the misfortune of introducing a problem that could be repro'd only on Mac and the question was whether to deal with this hack, or just use printfs. I used printfs, but I next time I can just debug on mac.

[steve-the-bayesian]

+1. This is a major pain point for me. Thanks to everyone who's working to fix it.

[tony-shi]

+1. BTW, will this issue be fixed/upgrade officially or we have to take a hack described above to enable generate debugging symbols?
Thanks!

[hlopko]

Hi all,

so I finally tried to look into this. Do you all agree that you can debug binaries on a mac just fine when you build them using bazel build -c dbg --spawn_strategy=local? Because that's what I'm seeing. I'm not trying to suggest disabling the sandbox is the fix for this issue, just that this workaround is not as complicated anymore as what was proposed in #2537 (comment).

Thank you!

[DavidGoldman]

rules_apple supports building dSYMs inside the linking action itself (like @haberman mentioned), although this means slowing down your critical path. If you ever build remotely you'll need dSYMs in order to debug unless your machine's paths matches the builders' paths exactly.

If you don't use dSYMs you'll need to disable sandboxing for the linking step, e.g. setting --strategy=ObjcLink=standalone and --strategy=CppLink=standalone. Like @haberman this wouldn't be a problem if the absolute paths to the .o files were still valid after the linking action finishes.

[wesleyw72]

Hi @hlopko , even with setting --spawn_strategy=local and -c dbg I still can’t get source level debugging. For example typing l <source file path> doesn’t work.

[hlopko]

@wesleyw72 I'm afraid 'll need a repro case for that, it should work AFAIK. Pls include Bazel version and all Bazel options you pass (don't forget the .bazelrc).

[wesleyw72]

@hlopko I think the issue I was hitting is that the cache was being hit, so changing the spawn_strategy wasn't making a difference as no actions were being spawned and thus the object files with the debugging symbols were missing (in a previously cleaned up sandbox). If I do a clean and then build with spawn_strategy=local then it does seem to be working. Thanks!

[nikhilm]

Trying to understand exactly what the deficiency here is, because it seems the macos_* rules from rules_apple can build dsyms.
Specifically, if one wants a dylib on macOS

cc_binary(
  name = "libfoo.dylib",
)

will not produce a dsym (nor will dwarf debug info be preserved in the dylib itself)

but

cc_library(
  name = "foo",
)

macos_dylib(
  name = "libfoo",
  deps = [":foo"],
  ...
)

when built with --apple_generate_dsym will generate valid dSYMs. I also see that wrapped_clang has a notion of DSYM_HINT_DSYM_PATH that is only set for objc targets in

bazel/tools/osx/crosstool/cc_toolchain_config.bzl

Line 5710 in fa1740d

"DSYM_HINT_DSYM_PATH=%{dsym_path}",

. Would extending that to also work for C/C++ executable actions be sufficient?

[haberman]

@nikhilm I am concern about the statically-linked binary case, not dylib.

In my view, this bug is fixed when I can:

1. build a cc_binary() in Bazel using -c opt or --copt=-g
2. run the binary in gdb/lldb/Instruments/etc
3. have full debugging symbols available.

When I do this today, I get no debugging symbols available.

[keith]

dSYMs are the solution for both of those cases. A workaround on macOS could actually even be to build a macos_command_line_application just for the dSYM support.

[bedbad]

As of right now I can confirm that when you build using "bazel build -c dbg --spawn_strategy=local" on Apple M1 ("darwin"), after clean, you can not find debug symbols in the binary.

[keith]

Since my previous comment the dSYM logic also applies to cc_binary targets now, can you pass --apple_generate_dsym and see if that fixes your issues?

[bedbad]

My bad. It does put the debug symbols in binary with the --spawn_strategy=local on Apple M1 Sillicon
[My eye glanced over the small "-c opt" flag further in the invocation].
Since this thread is still fairly active, I'd add the general method for debugging bazel flag issues like these which in my experience is most efficient (and get some bazel team feedback), since I had to resolve a good mass of bazel issues porting large build on Apple Sillicon M1 ("darwin" - arm64). Which are still newish and by far the fastest personal machine I built on out of all.

1. • run bazelisk ... with "-s" option (print the full command invocations).
     Identify the action of interest and its bazel type, note the full command invocation. and concrete arguments of interest. In this example "-g0", "-O2", "-DNDEBUG" (precluding clang debug symbols) and/or the wrapper/tool path bazel uses in it.
     Even if you try hard to pass the correct flags to bazel they are still be copied later in invocation, which in compiler case (not in bazel which looks to be inverse case) means they are actually ignored. (which was the case above)
2. Map these to the actual bazel configuration that runs in your case that you can find in ${output_base}/
   bazelisk info

...
output_base: /private/var/tmp/_bazel_im/b5fbef499f0aa54a50d4f7f63c872539
...

And search the actual code in places like ${output_base}/install/embedded_tools/tools/[ cpp| osx | ...]
Where your flags of interest originate from.
install/embedded_tools/tools/cpp/cc_toolchain_config.bzl:

   526	    if (ctx.attr.cpu == "darwin"):

   527	        default_compile_flags_feature = feature(
   528	            name = "default_compile_flags",
   529	            enabled = True,
   530	            flag_sets = [
   531	                flag_set(
   532	                    actions = [
   533	                        _ASSEMBLE_ACTION_NAME,
   534	                        _PREPROCESS_ASSEMBLE_ACTION_NAME,
   535	                        _LINKSTAMP_COMPILE_ACTION_NAME,
   536	                        _C_COMPILE_ACTION_NAME,
   537	                        _CPP_COMPILE_ACTION_NAME,
   538	                        _CPP_HEADER_PARSING_ACTION_NAME,
   539	                        _CPP_MODULE_COMPILE_ACTION_NAME,
   540	                        _CPP_MODULE_CODEGEN_ACTION_NAME,
   541	                        _LTO_BACKEND_ACTION_NAME,
   542	                        _CLIF_MATCH_ACTION_NAME,
   543	                    ],
   544	                    flag_groups = [
   545	                        flag_group(
   546	                            flags = [
   547	                                "-D_FORTIFY_SOURCE=1",
   548	                                "-fstack-protector",
   549	                                "-fcolor-diagnostics",
   550	                                "-Wall",
   551	                                "-Wthread-safety",
   552	                                "-Wself-assign",
   553	                                "-fno-omit-frame-pointer",
   554	                            ],
   555	                        ),
   556	                    ],
   557	                ),
   558	                flag_set(
   559	                    actions = [
   560	                        _ASSEMBLE_ACTION_NAME,
   561	                        _PREPROCESS_ASSEMBLE_ACTION_NAME,
   562	                        _LINKSTAMP_COMPILE_ACTION_NAME,
   563	                        _C_COMPILE_ACTION_NAME,
   564	                        _CPP_COMPILE_ACTION_NAME,
   565	                        _CPP_HEADER_PARSING_ACTION_NAME,
   566	                        _CPP_MODULE_COMPILE_ACTION_NAME,
   567	                        _CPP_MODULE_CODEGEN_ACTION_NAME,
   568	                        _LTO_BACKEND_ACTION_NAME,
   569	                        _CLIF_MATCH_ACTION_NAME,
   570	                    ],
   571	                    flag_groups = [flag_group(flags = ["-g"])],
   572	                    with_features = [with_feature_set(features = ["dbg"])],
   573	                ),
   574	                flag_set(
   575	                    actions = [
   576	                        _ASSEMBLE_ACTION_NAME,
   577	                        _PREPROCESS_ASSEMBLE_ACTION_NAME,
   578	                        _LINKSTAMP_COMPILE_ACTION_NAME,
   579	                        _C_COMPILE_ACTION_NAME,
   580	                        _CPP_COMPILE_ACTION_NAME,
   581	                        _CPP_HEADER_PARSING_ACTION_NAME,
   582	                        _CPP_MODULE_COMPILE_ACTION_NAME,
   583	                        _CPP_MODULE_CODEGEN_ACTION_NAME,
   584	                        _LTO_BACKEND_ACTION_NAME,
   585	                        _CLIF_MATCH_ACTION_NAME,
   586	                    ],
   587	                    flag_groups = [
   588	                        flag_group(
   589	                            flags = [
   
   590	                                "-g0",
   591	                                "-O2",
   592	                                "-DNDEBUG",

   593	                                "-ffunction-sections",
   594	                                "-fdata-sections",
   595	                            ],
   596	                        ),
   597	                    ],
   598	                    with_features = [with_feature_set(features = ["opt"])],
   599	                ),
   600	                flag_set(
   601	                    actions = [
   602	                        _LINKSTAMP_COMPILE_ACTION_NAME,
   603	                        _CPP_COMPILE_ACTION_NAME,
   604	                        _CPP_HEADER_PARSING_ACTION_NAME,
   605	                        _CPP_MODULE_COMPILE_ACTION_NAME,
   606	                        _CPP_MODULE_CODEGEN_ACTION_NAME,
   607	                        _LTO_BACKEND_ACTION_NAME,
   608	                        _CLIF_MATCH_ACTION_NAME,
   609	                    ],
   610	                    flag_groups = [flag_group(flags = ["-std=c++0x"])],
   611	                ),
   612	            ],
   613	        )

3. Understand the relevant code. Trace it upwards and so you can be sure what exactly bazel does in your case.
   For example, in Apple M1 debug case those arguments happen in darwin exclusively with_feature_set(features = ["opt"]) which categorically told me that "-c opt" is still somehow in my invocation(which was confusing because it looks like --copt)

This at least has the lowest guarantee on completion time which can be huge and has always improving learning curve. Although it can be much faster then try-balling from issues/SO for the particular complicated build which production bazel builds normally are. I don't really find that available documentation currently helps such issues much.

[constinit]

I can see debug symbols (including line numbers) when building with -c dbg --spawn_strategy=local on an M1 Mac with homebrew llvm.

Full command:

bazel  test --config=asan -c dbg --spawn_strategy=local //...

[fansehep]

I meet thie problem in my archlinux too.
bazel version: 5.3.1
archlinux version: 5.19.3
gcc version: 12.2.0
gdb version: 12.1
Even if i use -dbg or debug mode useless.
but seems the gdb find the symbols. but don't display.
The gdb tui display Reading symbols from bazel-bin/src/base/test/..., but don;t display.

[umap341]

On windows also facing same issue: (No debugging symbols found in bazel-bin/examples/application/application). Any solution to this?

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[chandlerc]

Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

Trying @bazelbuild/triage -- this is definitely still relevant and should be resolved. It is quite easy to reproduce.

[chandlerc]

Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

Trying @bazelbuild/triage -- this is definitely still relevant and should be resolved. It is quite easy to reproduce.

And as I suspected, I can't actually follow the steps described here: typing @bazelbuild/triage does nothing for me. It is incredibly infuriating to have a bot auto-close issues in an open source repository with instructions that no reporter can actually follow for keeping issues open. =[

[fmeum]

@bazelbuild/triage

I remember someone from the team saying that secret teams can still be mentioned even if autocomplete doesn't work for them.