Suggestion for more debug symbol folder file layouts (remotely-mounted system ones, debuginfod…)

[illwieckz]

Hi! The DEVELOPMENT.md file states:

The symbols file must named in one of three ways. The same fname as the binary (game.elf), the same name plus the .debug extension (game.elf.debug) or the same name but the .debug extension instead of the original one (game.debug). To make sure the binary and symbols file have been produced in the same build, Orbit checks that they have a matching build id.

This is very limiting, especially when using OrbitService on one computer and Orbit on another. For some reasons Orbit finds the symbols provided by Ubuntu Linux on /usr/lib/debug if the client is running on the same computer the profiled program runs, but if I run the profiled program with OrbitService running on another computer and I share the /usr/lib/debug folder of the remote computer over the network to be mounted on the local computer running Orbit, and add that mount point to the list of folder to look debug symboles for, it doesn't work.

The file layout is the same as /usr/lib/debug, so here is how it is on the remote computer:

/usr/lib/debug/.build-id/04/6a33a6ce5fcf079cda4518fc4709613aeb2529.debug
/usr/lib/debug/.build-id/04/91a649034f20c664c5ae69b5f48b4d20eed60b.debug
/usr/lib/debug/.build-id/11/2149accfe72667572e7821ace517b326f6ff90.debug
/usr/lib/debug/.build-id/11/a8acf3e985b29604eb782fde22b57fb6167068.debug
/usr/lib/debug/.build-id/1a/e0cb77020c59b410aa4719a7a23651221635b3.debug

Here is how it is if machine:/usr/lib/debug is mounted as /data/machine/debug:

/data/machine/debug/.build-id/04/6a33a6ce5fcf079cda4518fc4709613aeb2529.debug
/data/machine/debug/.build-id/04/91a649034f20c664c5ae69b5f48b4d20eed60b.debug
/data/machine/debug/.build-id/11/2149accfe72667572e7821ace517b326f6ff90.debug
/data/machine/debug/.build-id/11/a8acf3e985b29604eb782fde22b57fb6167068.debug
/data/machine/debug/.build-id/1a/e0cb77020c59b410aa4719a7a23651221635b3.debug

When the build id is 1ae0cb77020c59b410aa4719a7a23651221635b3 the filename is <symbol_folder>/.build-id/1a/e0cb77020c59b410aa4719a7a23651221635b3.debug, not <symbol_folder>/<exe_name>.debug, and there is probably no way to guess back the original executable name to please Orbit.

The second need is that such system's debug folder also provide debug symbol files with such file layout:

/usr/lib/debug/.dwz/x86_64-linux-gnu/libglib2.0-0.debug
/usr/lib/debug/.dwz/x86_64-linux-gnu/libxslt1.1.debug
/usr/lib/debug/.dwz/x86_64-linux-gnu/libxvmc1.debug
/usr/lib/debug/.dwz/x86_64-linux-gnu/libyelp0.debug
/usr/lib/debug/.dwz/x86_64-linux-gnu/libzvbi0.debug

So a mounted folder from a remote computer may be:

/data/machine/debug/.dwz/x86_64-linux-gnu/libglib2.0-0.debug
/data/machine/debug/.dwz/x86_64-linux-gnu/libxslt1.1.debug
/data/machine/debug/.dwz/x86_64-linux-gnu/libxvmc1.debug
/data/machine/debug/.dwz/x86_64-linux-gnu/libyelp0.debug
/data/machine/debug/.dwz/x86_64-linux-gnu/libzvbi0.debug

When the executable name is libglib2.0-0 the filename is <symbol_folder>/.dwz/<arch-triple>/libglib2.0-0.debug, not <symbol_folder>/<exe_name>.debug, this one may be easy to workaround.

The third need, is the need for the debuginfod layout, this is a service provided by some distros like Ubuntu, when running a program on GDB, GDB automatically download the symbols for the third-party loaded libraries from a debuginfod server, this is very convenient.

Here is an example of the debuginfod file layout:

/home/illwieckz/.cache/debuginfod_client/1d72a8007f763be42a9a7c5d63be49d19b51488e/debuginfo
/home/illwieckz/.cache/debuginfod_client/69639d0edb485049e48a8257da67e1724823c52d/debuginfo
/home/illwieckz/.cache/debuginfod_client/7bd8992ae8f799489b7cd5c47395348f14d4fde1/debuginfo
/home/illwieckz/.cache/debuginfod_client/dceee56a30bfe4dbbd373093772ecc4b3e09ae03/debuginfo
/home/illwieckz/.cache/debuginfod_client/ffe88b55e233eb6eff03586e0818d55e769e79b2/debuginfo

With /home/illwieckz/.cache being XDG_CACHE_HOME as defined by the XDG specification, more precisely ${XDG_CACHE_HOME:-${HOME}/.cache.

When the build-id is ffe88b55e233eb6eff03586e0818d55e769e79b2 the file name is <symbol_folder>/ffe88b55e233eb6eff03586e0818d55e769e79b2/debuginfo, , not <exe_name>.debug, and there is probably no way to guess back the original executable name to please Orbit.

So I suggest to add the support for <symbol_folder>/.build-id/<bu/ild_id>.debug, <symbol_folder>/.dwz/<arch-triple>/<exe_name>.debug and <symbol_folder>/<build_id>/debugino file names in Orbit debug symbol folders.

An extra option would be to also look for <symbol_folder>/<build_id>/debugino, this way a single /data/machine/debug folder exposing a remote computer filesystem would be able to provide both (according to the fact the remote filesystem would be configured to store everything in a single parent folder):

/data/machine/debug/.build-id/04/6a33a6ce5fcf079cda4518fc4709613aeb2529.debug
/data/machine/debug/.dwz/x86_64-linux-gnu/libglib2.0-0.debug
/data/machine/debug/debuginfod_client/ffe88b55e233eb6eff03586e0818d55e769e79b2/debuginfo

Thank you for this tool which is awesome and it would become much more awesome this way! 😍️

[illwieckz]

I notice this other issue for a similar problem on Gentoo:

• Support split debug symbols without build-id (gentoo-style) #4832

On that system the path is:

/usr/lib/debug/lib64/libc.so.6.debug
/usr/lib/debug/lib/libc.so.6.debug

So an additional debug folder look-up should also look for <symbol_folder>/lib*/<exe_name>.debug files.

[fche]

Note that for debuginfod model of usage, the profiler can spawn a call to debuginfod-find(1) (or make the equivalent http GET itself) to attempt to fetch debuginfo on the fly. It is not stuck looking only for previously received content.

https://sourceware.org/elfutils/Debuginfod.html