Implement optional pretty printed stacktraces #2420

codablock · 2018-11-05T12:15:28Z

This solves something that is very much on top of my "Why I hate C++" list. I lost so much time already hunting exceptions/segfaults which only happened sporadically in tests. With this PR, you can add --enable-stacktraces to ./configure, which will give you pretty stacktraces on nearly all kinds of crashes that might happen. It also adds better default handling for catched unknown (catch(...)) exceptions.

This works on Linux and Windows (only MinGW based, native and wine). On Linux it uses backtrace() from execinfo.h to gather the stack trace and StackWalk64 on Windows. It then uses libbacktrace to extract symbol information. It is required to have debug information enabled, so configure.ac will add -g to the compiler flags, but still build with optimization (if --enable-debug is not given at the same time).

For all this to work, I had to "hack" into the C++ ABI by using -Wl,-wrap. It wraps a few of the ABI methods and gathers the stacktrace before actually doing the exception handling. GetExceptionStacktraceStr is then able to retrieve the stacktrace from a catched exception.

This method only works on GCC. For Clang based builds, I implemented dlsym(RTLD_NEXT, ...) based hooks into the C++ standard libs. This is untested currently as I don't have a Mac to test this. Would be happy if someone could test this and give feedback (@UdjinM6 @nmarley maybe). At least it's able to compile...so I have some hope it works :)

MacOS support is not complete yet, as it requires some changes to the build system. We specifically need to generate .dSYM bundles which contain the necessary debug symbols. Until then, --enable-stacktraces will produce many libbacktrace errors in the debug output and not show any useful information about stacktraces.

codablock · 2018-11-05T12:21:37Z

FYI, if this turns out to work well enough, we could also consider adding tracing of stacktraces to the deadlock detector. Depends on how slow the stacktrace gathering is in the end...

codablock · 2018-11-05T15:07:35Z

Another option is to always build with stacktrace support enabled and make it a runtime option. It would then be disabled by default in a way that it does not create any overhead (bail out immediately from wrapped ABIs)

UdjinM6 · 2018-11-06T05:53:52Z

macos 10.14.1 (18B75)

> ./src/test/test_dash 
Running 250 test cases...
libbacktrace error: 0 - libbacktrace could not find executable to open
libbacktrace error: -1 - failed to read executable information
libbacktrace error: -1 - failed to read executable information
libbacktrace error: -1 - failed to read executable information
libbacktrace error: -1 - failed to read executable information
#### std::terminate() called, aborting ####
#### UNCAUGHT EXCEPTION ####
Exception: type=, what="whoops"
   0#: (0x00000000) <unknown-file> - ???
   1#: (0x00000000) <unknown-file> - ???
   2#: (0x00000000) <unknown-file> - ???
   3#: (0x00000000) <unknown-file> - ???
   4#: (0x00000000) <unknown-file> - ???
unknown location:0: fatal error: in "evo_dip3_activation_tests/dip3_activation": signal: SIGABRT (application abort requested)
test/test_dash.cpp:83: last checkpoint
Segmentation fault: 11

codablock · 2018-11-09T16:13:18Z

@UdjinM6 Can you try it again if you find time?

UdjinM6 · 2018-11-09T16:54:16Z

> ./src/dashd --regtest
4559648916, , 0, 
140734756526909, , 0, 
0, , 0, 
4559656182, , 0, 
140734756561721, , 0, 
140734756573863, , 0, 
140734756557893, , 0, 
sis=7
lstr=<unknown-file>
lstr=<unknown-file>
lstr=<unknown-file>
lstr=<unknown-file>
lstr=<unknown-file>
lstr=<unknown-file>
lstr=<unknown-file>
s2= 0#: (0x10FC6B894) <unknown-file> - ???

s2= 1#: (0x7FFF5D2B9B3D) <unknown-file> - ???

s2= 2#: (0x00000000) <unknown-file> - ???

s2= 3#: (0x10FC6D4F6) <unknown-file> - ???

s2= 4#: (0x7FFF5D2C2339) <unknown-file> - ???

s2= 5#: (0x7FFF5D2C52A7) <unknown-file> - ???

s2= 6#: (0x7FFF5D2C1445) <unknown-file> - ???

#### signal Segmentation fault: 11 ####
   0#: (0x10FC6B894) <unknown-file> - ???
   1#: (0x7FFF5D2B9B3D) <unknown-file> - ???
   2#: (0x00000000) <unknown-file> - ???
   3#: (0x10FC6D4F6) <unknown-file> - ???
   4#: (0x7FFF5D2C2339) <unknown-file> - ???
   5#: (0x7FFF5D2C52A7) <unknown-file> - ???
   6#: (0x7FFF5D2C1445) <unknown-file> - ???
Abort trap: 6

codablock · 2018-11-10T06:52:27Z

Turned out Mac support was never merged into libbacktrace (ianlancetaylor/libbacktrace#2). But the necessary code is used by the Rust devs in https://github.com/rust-lang-nursery/libbacktrace/tree/rust-snapshot-2018-05-22 (they were also the ones who created the initial PR). I changed backtrace.mk to use this repo instead of the upstream repo until Mac support is officially supported by the upstream version. @UdjinM6 If you find time...one last try...if it still doesn't work I give up for now.

UdjinM6 · 2018-11-10T07:41:22Z

> ./src/dashd --regtest
exeFileName: /Developer/Projects/crypto/dash_develop/./src/dashd
libbacktrace error: -1 - no debug info in Mach-O executable
exeFileName: /Developer/Projects/crypto/dash_develop/./src/dashd
libbacktrace error: -1 - no debug info in Mach-O executable
exeFileName: /Developer/Projects/crypto/dash_develop/./src/dashd
libbacktrace error: -1 - no debug info in Mach-O executable
exeFileName: /Developer/Projects/crypto/dash_develop/./src/dashd
libbacktrace error: -1 - no debug info in Mach-O executable
exeFileName: /Developer/Projects/crypto/dash_develop/./src/dashd
libbacktrace error: -1 - no debug info in Mach-O executable
exeFileName: /Developer/Projects/crypto/dash_develop/./src/dashd
libbacktrace error: -1 - no debug info in Mach-O executable
exeFileName: /Developer/Projects/crypto/dash_develop/./src/dashd
libbacktrace error: -1 - no debug info in Mach-O executable
exeFileName: /Developer/Projects/crypto/dash_develop/./src/dashd
libbacktrace error: -1 - no debug info in Mach-O executable
#### signal Segmentation fault: 11 ####

Abort trap: 6

codablock · 2018-11-26T19:35:40Z

So, after some investigation it looks like stacktraces on Mac will require more work in the build system. We need to manually generate .dSYM bundles with debug information to make make libbacktrace work on Mac OS. I'll leave it as it is now, simply don't use --enable-stacktraces on Mac for now.

A build is running atm that does a test crash. I'll post a link to it later.

This is currently only useful to extract symbols. It fails to gather stacktraces when compiled with MinGW, so we can only use it to get symbol information from a stack trace which we gathered outside of libbacktrace.

This is a hack and should only be used for debugging. It works by wrapping the C++ ABI __wrap___cxa_allocate_exception. The wrapper records a backtrace and stores it in a global map. Later the stacktrace can be retrieved with GetExceptionStacktraceStr. This commit also adds handlers to pretty print uncaught exceptions and signals.

codablock · 2019-02-19T06:46:41Z

Rebased this branch and reworked a few things. The last Travis build contains a few test crashes so that you can see how stacktraces look like.

Unfortunately, the stack traces get all mixed up when walking too deep, but this is something I currently have no good solution for. Reason is that the backtrace() call done in GetStackFrames returns too many stack frames, so that it goes beyond the entry point of the process/thread.

I've disabled stacktraces on the win32/win64 Travis builds for now, as the added debug info results in binaries with multiple hundred MBs of size, causing the linker to take ages (which times out on Travis). If future mingw versions turn out to perform better, we might enable stacktraces for win32/win64 on Travis.

MacOS is not supported for now. I've also updated the PR description to make this clear.

I'm going to remove the WIP status now, meaning that I'm happy to see reviews and eventually ACKs :)
I'll remove the test crashes when ACKs come in.

UdjinM6 · 2019-02-19T14:03:03Z

Looks good 👍

Otherwise the code at the bottom is never executed when nodes crash, leading to no output of debug.log files on Travis.

codablock · 2019-02-21T08:58:55Z

Removed the test crashes and force-pushed

UdjinM6 · 2019-02-21T11:29:45Z

Shouldn't we also drop TestCrash, DoCrash etc.?

codablock · 2019-02-21T11:36:27Z

@UdjinM6 Yepp, should have removed them, which I did now.

UdjinM6

👍

utACK

This reverts commit 48d92f1.

codablock force-pushed the pr_stacktraces branch from c95580e to 9a796ce Compare November 9, 2018 15:23

codablock force-pushed the pr_stacktraces branch from 9a796ce to 0e31cc3 Compare November 9, 2018 16:26

codablock force-pushed the pr_stacktraces branch from 6ac9e81 to 9ad6b0b Compare November 10, 2018 07:15

codablock force-pushed the pr_stacktraces branch from 094e71f to 939ed0d Compare November 26, 2018 19:32

codablock force-pushed the pr_stacktraces branch from 939ed0d to aa37167 Compare November 26, 2018 20:35

codablock changed the title ~~Implement optional pretty printed stacktraces~~ [WIP] Implement optional pretty printed stacktraces Nov 27, 2018

codablock force-pushed the pr_stacktraces branch from c9a9159 to c876ec7 Compare November 28, 2018 11:22

codablock mentioned this pull request Dec 11, 2018

Bail out from update methods in MasternodeList when shutdown is requested #2551

Merged

codablock force-pushed the pr_stacktraces branch from c876ec7 to 0a7c6c1 Compare January 11, 2019 06:39

codablock force-pushed the pr_stacktraces branch 2 times, most recently from c1bbc1d to c3008c5 Compare January 28, 2019 13:27

codablock added 2 commits February 19, 2019 06:08

Add libbacktrace to depends

7dd0565

This is currently only useful to extract symbols. It fails to gather stacktraces when compiled with MinGW, so we can only use it to get symbol information from a stack trace which we gathered outside of libbacktrace.

Add -mbig-obj to CXXFLAGS for MinGW builds

29759d3

codablock force-pushed the pr_stacktraces branch from c3008c5 to c81b1e4 Compare February 19, 2019 05:19

codablock added 4 commits February 19, 2019 07:07

Register exception translators to pretty print exceptions in unit tests

0dbdf00

Use GetPrettyExceptionStr for all unhandled exceptions

ffc6bd5

Use --enable-stacktraces in CI for linux32/linux64

a99d259

codablock force-pushed the pr_stacktraces branch from c81b1e4 to 2c3a035 Compare February 19, 2019 06:08

codablock changed the title ~~[WIP] Implement optional pretty printed stacktraces~~ Implement optional pretty printed stacktraces Feb 19, 2019

UdjinM6 added this to the 14.0 milestone Feb 19, 2019

Catch and print python exceptions when stopping nodes

1a4cac0

Otherwise the code at the bottom is never executed when nodes crash, leading to no output of debug.log files on Travis.

codablock force-pushed the pr_stacktraces branch from 2c3a035 to 1a4cac0 Compare February 21, 2019 08:57

Remove now unneeded/unused TestCrash methods

ad0b522

UdjinM6 approved these changes Feb 21, 2019

View reviewed changes

UdjinM6 merged commit 48d92f1 into dashpay:develop Feb 21, 2019

nmarley added a commit to nmarley/dash that referenced this pull request Sep 16, 2019

Revert "Implement optional pretty printed stacktraces (dashpay#2420)"

ea6c5e2

This reverts commit 48d92f1.

nmarley added a commit to nmarley/dash that referenced this pull request Sep 16, 2019

Revert "Implement optional pretty printed stacktraces (dashpay#2420)"

6b159e8

This reverts commit 48d92f1.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement optional pretty printed stacktraces #2420

Implement optional pretty printed stacktraces #2420

codablock commented Nov 5, 2018 •

edited

Loading

codablock commented Nov 5, 2018

codablock commented Nov 5, 2018

UdjinM6 commented Nov 6, 2018

codablock commented Nov 9, 2018

UdjinM6 commented Nov 9, 2018

codablock commented Nov 10, 2018

UdjinM6 commented Nov 10, 2018

codablock commented Nov 26, 2018

codablock commented Feb 19, 2019 •

edited

Loading

UdjinM6 commented Feb 19, 2019

codablock commented Feb 21, 2019

UdjinM6 commented Feb 21, 2019

codablock commented Feb 21, 2019

UdjinM6 left a comment

Implement optional pretty printed stacktraces #2420

Implement optional pretty printed stacktraces #2420

Conversation

codablock commented Nov 5, 2018 • edited Loading

codablock commented Nov 5, 2018

codablock commented Nov 5, 2018

UdjinM6 commented Nov 6, 2018

codablock commented Nov 9, 2018

UdjinM6 commented Nov 9, 2018

codablock commented Nov 10, 2018

UdjinM6 commented Nov 10, 2018

codablock commented Nov 26, 2018

codablock commented Feb 19, 2019 • edited Loading

UdjinM6 commented Feb 19, 2019

codablock commented Feb 21, 2019

UdjinM6 commented Feb 21, 2019

codablock commented Feb 21, 2019

UdjinM6 left a comment

Choose a reason for hiding this comment

codablock commented Nov 5, 2018 •

edited

Loading

codablock commented Feb 19, 2019 •

edited

Loading