Coredump possibilities in AppScope

[michalbiesek]

Rationale

This issue here describes the possibility of AppScope finding the root cause of crashing the process.

Resolution details

• investigate the possibility of generating a core dump using the userspace mechanism
• create the core dump using signal-safe API - so the core dump can be generated on the signal handler
• the solution should support various environments x86_64/aarch64 glibc/musl
• solution should support container env as well

Current status

• I was able to generate correct core dump on x86_64 host env (glibc) on x86_64 container env (glibc)
• I was able to generate correct core dump on aarch64 host env (glibc) on aarch64 container env (glibc)

Disclaimer:

• on aaarch64 we broke the libscope.so on musl because libcoredumper code wants to use getcontext symbol which is missing in musl including our internal library as well.
• The getcontext implementation is provided and used in another library used by us in libunwind Therefore we in libcoredumper.a code we call the function from libunwind.
• I saw that in libunwind upstream repo there was a fix for getcontext implementation see details - therefore I have updated the libunwind src code in contrib as well

[michalbiesek]

Status:
Current state of Pull Request 1289 provides a support for working coredump feature on:

• glibc x86_64

• glibc aarch64

• With proper changing the getcontext I do not broke libscope.so build anymore.

• Extending the SCOPE_ERROR_SIGNAL_HANDLER values was added which allows to on error signal received:

  • generate information in log about the backtrace
  • generate core dump
  • both

Next steps:
Currently, I see that generation of core dump on musl (both x86_64 and aarch64) is broken - the core dump is not usable for a user - we do not even see the symbol which comes from libscope.so - I need to investigate it why
Starting from comparing the core generated by us and by the kernel to see differences. ##1303 tracks this next step.

[michalbiesek]

Current status:
After a discussion with @jrcheli to wrap up this issue. I plan to split the current work presented into 4 Pull Requests which all together will be addressed as core dump functionality on glibc system. Splitting this functionality into 4 Pull Requests will be much easier to merge & review.

• Bump version of libunwind - reference [#1279 1/4] Update libunwind version #1294
• Add coredumper to contrib code - reference [#1279 2/4] Extend contrib code with coredumper #1295
• Perform necessary changes in build system to integrate coredumper e.g. workaround for missing getcontext implementation - [#1279 3/4] Integrate coredumper with scope codebase #1296
• Extend Appscope library code to use coredumper API to generate core dump in libscope.so - [#1279 4/4] Generate core dump in libscope.so #1299

[jrcheli]

The work here to be able to create a coredump from libscope.so has been added here. I've verified that it works well on glibc-based applications. #1303 captures that additional work will be required to support musl-based applications.

I'm closed this issue manually, since it's been merged into the release/1.3 branch.