Distribute official builds with full debugging symbols

[Calinou]

Describe the project you are working on

The Godot editor 🙂

Describe the problem or limitation you are having in your project

For the sake of small file sizes and optimization, official builds are stripped from their debugging symbols and have link-time optimization enabled. Unfortunately, LTO is currently incompatible with accurate debugging symbols (like some optimizations found at the -O2 level in general).

While optimized builds are fast, they're impractical to use for debugging as their backtraces aren't useful.

I currently distribute builds with full debugging information in this repository, but an official solution will likely scale better over time.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

To ease troubleshooting of crashes and other difficult-to-diagnose issues, we should distribute official builds that are built with the debug target and contain full debugging symbols, especially for Windows where compiling is more difficult.

Windows debug builds should preferably be compiled with MSVC so people can use the Visual Studio Debugger or WinDbg. (It would be possible to build them with MinGW and tell people to use gdb, but that might not be accepted as much.)

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams

Create and distribute builds with debugging symbols on the official Godot download mirrors.

These builds could be referenced on the website under a single link for all platforms on the Download page. (Most people won't need those builds, so these shouldn't be made too prominent.)

Since these builds are slower and significantly larger, these should only be used for debugging purposes. There should be a disclaimer stating these shouldn't be used in a production environment.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No, as this is related to official build distribution.

Is there a reason why this should be core and not an add-on in the asset library?

This is about making official builds.

---

See also #1341.

[fire]

Here are some flags for exporting llvm mingw with pdbs.

use_mingw=yes use_llvm=yes use_lld=yes use_thinlto=yes LINKFLAGS=-Wl,-pdb= CCFLAGS='-g -gcodeview' debug_symbols=no

Hope this helps.

https://github.com/mstorsjo/llvm-mingw

[likeich]

Would these work with android to provide actual stack traces for native code crashes?

[Calinou]

Would these work with android to provide actual stack traces for native code crashes?

Yes, although you'll want to separate the debug symbols from the binary for use in production. Google Play has a tool that lets you "decipher" crash backtraces submitted by users using local debug symbols files.

[likeich]

Would these work with android to provide actual stack traces for native code crashes?

Yes, although you'll want to separate the debug symbols from the binary for use in production. Google Play has a tool that lets you "decipher" crash backtraces submitted by users using local debug symbols files.

Thanks for the response. Is there any documentation for how to do this? I released an app today and I'm having a ton of native crashes that I need to figure out.

[Calinou]

Thanks for the response. Is there any documentation for how to do this? I released an app today and I'm having a ton of native crashes that I need to figure out.

Unfortunately, this is not trivial to do. To my knowledge, Godot's buildsystem currently doesn't provide the required facilities to do this out of the box.

@m4gr3d said he'd look at implementing this a while ago, but he didn't have time to do it.

[jasonwinterpixel]

Can you provide any more insight on what needs to be done in order to generate debug symbols for godot?

[Calinou]

Can you provide any more insight on what needs to be done in order to generate debug symbols for godot?

• target=debug and target=release_debug builds contain debug symbols by default.
• target=release builds don't contain debug symbols by default, but you can force generation of debug symbols by using target=release debug_symbols=yes.
• Do not run strip on the compiled binaries, as doing so removes their debug symbols.
• On Windows, use MSVC to build Godot instead of MinGW, as the crash handler currently doesn't work with MinGW.

[jknightdoeswork]

We are compiling Godot ourselves, and would like clear information on how to get symbols uploaded into Google play properly. I've been struggling with this for years.

[jknightdoeswork]

Finally figured it out. The android build system is slightly awkward because it packages godot into a .aar, and then uses that .aar during the export phase. When you compile with symbols, those symbols dont end up in the .aar in res://android, so they dont end up in the final build. You can fix this by preventing all stripping during the compilation of the export templates. The symbols are ultimately stripped by the creation of the .aab on export.

Compile android templates always with symbols:

godot/platform/android/detect.py

env.Append(CCFLAGS=["-g"]) # always with debug symbols and build id

Package the unstripped libs into the .aar:

gradlew generateGodotTemplates -PdoNotStrip=true

Now when you export, the gradle command that does the export ends up stripping the symbols and generating the appropriate .dbg file in the resulting .aab.

The changes that I propose are this:

1. Android compilation should either always compile with debug symbols or should at least support debug_symbols=yes
   Maybe I'm missing something, but as far as I can tell, there is no support for compiling release builds with symbols in godot/android/platform/detect.py.

2. Android 'template generation' ie: gradlew generateGodotTemplates should either always use the -PdoNotStrip=true argument or it should be documented on the compile for android page (along with information that symbols will be stripped out by the final export process)

I recommend going with the 'always compile with symbols' and 'always leave symbols in the .aar template', because then all godot games uploaded to the app store will have debug symbols in their traces automatically by google.

[m4gr3d]

The changes that I propose are this:

1. Android compilation should either always compile with debug symbols or should at least support debug_symbols=yes
   Maybe I'm missing something, but as far as I can tell, there is no support for compiling release builds with symbols in godot/android/platform/detect.py.
2. Android 'template generation' ie: gradlew generateGodotTemplates should either always use the -PdoNotStrip=true argument or it should be documented on the compile for android page (along with information that symbols will be stripped out by the final export process)

I recommend going with the 'always compile with symbols' and 'always leave symbols in the .aar template', because then all godot games uploaded to the app store will have debug symbols in their traces automatically by google.

For aab generation, it's worth considering since the Google Play Store will ensure that only the necessary data is downloaded to the user's device. For apk generation on the other hand, we need to ensure there's mechanism to strip out the debug symbols within the Godot editor since otherwise it would make the apk larger for the user to download.

@jknightdoeswork Can you provide an estimate of the size difference for both the aab and apk without debug symbols, and the aab and apk with debug symbols?

[Chaosed0]

I would like to put forward that debug symbols should be distributed even with release builds of the engine. It appears that the build system already supports exporting symbol files (pdb in Windows) separate from the executable, and that should not inflate the binary size.

If this PR gets in (and I'm really hoping it does), then those symbol files can be used to symbolicate native crashes that occur within the engine. godotengine/godot#56014

[hhyyrylainen]

With breakpad the symbol files need to be pre-processed into the format that stackwalk tool understands. If breakpad was added to the official Godot releases then distributing the processed .sym files would be enough for developers to be able to extract callstacks from breakpad created crash dump files they could ask players to send (or create an automated tool to send crash dump files, this is what I've done for Thrive with a custom Launcher program).

[tobydjones]

I am experiencing a Godot editor crash #94255 on a Ubuntu VM.

I don't really want to have to learn how to compile Godot to debug this issue, especially as, reading this thread, it doesn't look like compiling with debug symbols is particularly easy!

@Calinou could you possibly add 4.2.2 to your repository please?
Or does anyone else have a debug version of 4.2.2 for Ubuntu I could download please?

[Calinou]

I might do that in the future but I can't guarantee it due to time constraints.

it doesn't look like compiling with debug symbols is particularly easy!

Compiling with debug symbols is the same as usual, you just need to append debug_symbols=yes to the SCons command line.

[huwpascoe]

Compiling with debug symbols is the same as usual, you just need to append debug_symbols=yes to the SCons command line.

If it's that easy then what's preventing this issue being resolved exactly?

[Calinou]

If it's that easy then what's preventing this issue being resolved exactly?

It's mostly about modifying https://github.com/godotengine/build-containers and https://github.com/godotengine/godot-build-scripts to keep debugging symbols, and ensuring the deploy scripts upload them to TuxFamily and GitHub Releases. This is work that needs to be coordinated with release managers.

There are also some caveats due to our use of MinGW for official Windows builds. The debug symbols will not work with the Visual Studio debugger or WinDbg out of the box; you'll need to run an external tool to convert the DWARF debugging symbols to PDB.