[3.x] Crash handler: Use print_error to include backtrace in logs

[akien-mga]

I'm a bit surprised that we never thought about changing this until now, as it's a recurring hurdle for users that even with logging enabled, they won't get the actual output of our crash handler. This should now be fixed.

This doesn't necessarily mean that the output will be super useful though, as unless you provide an unstripped debug build to end users, the backtrace will be mangled or just ???. I also think it's not working well on Windows when using MSVC (or when using MinGW, I don't remember). Anyway, this was a low hanging fruit to improve things a bit.

master PR: #60782

• Fixes Engine crashes without leaving a log #47379.

---

Only tested on Linux so far. Help welcome to test macOS and Windows, and notably to see what's the output for both Windows MSVC and MinGW.

Test project (triggers a crash using OS.crash(), also handles NOTIFICATION_CRASH from script). It's configured to write the log directly in the project folder / next to the exported binary.
CrashTest.zip

[bruvzg]

print_error will work only if OS logger is initialized and functional, I guess crash can happen when it's not. But it is probably fine for most cases.

as unless you provide an unstripped debug build to end users, the backtrace will be mangled or just ???.

Might be better to build with separate symbols (Scons flag should be there) and have it as an optional download, instead of providing unstripped binaries.

[akien-mga]

print_error will work only if OS logger is initialized and functional, I guess crash can happen when it's not. But it is probably fine for most cases.

Yeah the crash handlers already check that OS::get_singleton() is valid - but that doesn't necessarily mean that the logger itself is. If we find that it's a problem, this could likely be reworked further so that print methods have a graceful handling of invalid logger state (e.g. to fall back on a simple fprintf to stdout/stderr).

as unless you provide an unstripped debug build to end users, the backtrace will be mangled or just ???.

Might be better to build with separate symbols (Scons flag should be there) and have it as an optional download, instead of providing unstripped binaries.

Yeah this requires some more testing and documentation, I'm not sure the workflow is very well oiled for this. That's something we should aim to improve IMO, currently users are a bit powerless to get information on remote crashes from their players.

@bruvzg BTW if you want to poke at improving the Windows crash handler, reading it I assume it only works with MSVC as it depends heavily on Win32 APIs. Might be nice to have another one used for MinGW that would likely work similarly to the Linux/macOS one (I'm speculating, I haven't actually tested this on Windows recently).

[bruvzg]

BTW if you want to poke at improving the Windows crash handler, reading it I assume it only works with MSVC as it depends heavily on Win32 APIs

It will probably work, but debug symbols format might be incompatible, I'll check it.

[bruvzg]

Seems like crash handler can be enabled for MinGW build with a minimal changes, but debug symbols are incompatible, so it's will either require including some sort of custom DWARF information parser (https://github.com/ssbssa/dwarfstack, but lindwarf is LGPL). Or executable should be preprocessed with a tool like https://github.com/rainers/cv2pdb to convert symbols.

[bruvzg]

Also, Breakpad/Crashpad doesn't seem to support MinGW either, at least out of the box. But sentry fork seems to have it add - https://github.com/getsentry/crashpad/blob/getsentry/README.getsentry.md

[akien-mga]

Tested the Windows/MSVC build from the PR's CI on Linux with Wine, it seems to work well too, the backtrace ends up in the godot.log.

As I've noticed in several bug reports, the Windows backtraces often seem to be weirdly hijacked by embree's TaskScheduler, that's also something we should look into (maybe it performs better with actual debug symbols). But that's outside the scope of this PR.

[bruvzg]

Tested on Windows/MSVC and macOS, stack trace gets to the file.