Build failure with LTO (`drti.c:181:(.text.startup+0x21c): undefined reference to `__SUNW_dof'`)

[thesamesam]

A bunch of distros now build with LTO by default, although Gentoo doesn't (yet?). We should try make that work, or if we really have to, pass -fno-lto for now in the build system.

We currently explode with:

x86_64-pc-linux-gnu-gcc -O3 -march=native -mtls-dialect=gnu2 -flto=jobserver -pipe -fcf-protection=none -fdiagnostics-color=always -fdiagnostics-urls=never -frecord-gcc-switches -Wa,-O2 -Wa,-mtune=znver2 -Wstrict-aliasing -Wfree-nonheap-object -Werror=lto-type-mismatch -Werror=strict-aliasing -Werror=odr -Wstrict-aliasing -Wfree-nonheap-object -Werror=lto-type-mismatch -Werror=strict-aliasing -Werror=odr -Wbuiltin-declaration-mismatch -ggdb3 -Wformat -Wformat-security -Waddress -Warray-bounds -Wfree-nonheap-object -Wint-to-pointer-cast -Wmain -Wnonnull -Wodr -Wpointer-to-int-cast -Wreturn-type -Wsizeof-pointer-memaccess -Wstrict-aliasing -Wstring-compare -Wuninitialized -Wmaybe-uninitialized -Wstringop-overflow -Wstringop-overread -Wstringop-truncation -Wdangling-pointer -Wformat-overflow -Wformat-truncation -Wuse-after-free -Wvarargs -fuse-ld=bfd -std=gnu99 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_DT_VERSION=\"2.0.1\" -Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs -flto=jobserver -fuse-ld=mold -Wl,--defsym=__gentoo_check_ldflags__=0 -ggdb3 -Wl,--undefined-version -fuse-ld=bfd -o /var/tmp/portage/dev-debug/dtrace-9999/work/dtrace-9999/build/usdt-tst-forker /var/tmp/portage/dev-debug/dtrace-9999/work/dtrace-9999/build/test-triggers--usdt-tst-forker.o /var/tmp/portage/dev-debug/dtrace-9999/work/dtrace-9999/build/test-triggers--usdt-tst-forker-prov.o -L/var/tmp/portage/dev-debug/dtrace-9999/work/dtrace-9999/build
/usr/lib/gcc/x86_64-pc-linux-gnu/15/../../../../x86_64-pc-linux-gnu/bin/ld.bfd: /var/tmp/portage/dev-debug/dtrace-9999/temp/ccVplKxz.ltrans0.ltrans.o: in function `dtrace_dof_init':
/var/tmp/portage/dev-debug/dtrace-9999/work/dtrace-9999/libdtrace/drti.c:181:(.text.startup+0x21c): undefined reference to `__SUNW_dof'
/usr/lib/gcc/x86_64-pc-linux-gnu/15/../../../../x86_64-pc-linux-gnu/bin/ld.bfd: /var/tmp/portage/dev-debug/dtrace-9999/temp/ccVplKxz.ltrans0.ltrans.o: in function `main':
/var/tmp/portage/dev-debug/dtrace-9999/work/dtrace-9999/test/triggers/usdt-tst-forker.c:19:(.text.startup+0x3cc): undefined reference to `__dtrace_forker___fire'
collect2: error: ld returned 1 exit status
make: *** [Makerules:31: /var/tmp/portage/d

I've not looked into it yet.

[nickalcock]

This is very much something we've never tried before -- but it should nearly work. The only thingss the USDT mechanism (that's failing here) absolutely require are that we can identify the location of references to specific symbols at link time and patch them into NOPs, recording their (section-relative) offsets in the DOF so we can put probes there later. Nothing there requires non-LTO that I know of, but the usdt-tst-forker error suggests that something is being thrown awry in this process. Probably the problem is that under LTO the actual compilation isn't done until link time, but when USDT is in use, dtrace -G modifies the input object files before link time and requires them to contain actual final references to USDT probe points. An object file containing no object code and nothing but IR to be later consumed by the compiler isn't going to be good enough.

The right fix for this is probably for DTrace to contribute a linker plugin so that it can get into the linking process after LTO and run dtrace -G for you, which would have the major advantage of making building programs containing USDT probes just like building normal programs: no special dtrace -G stage. But I have absolutely no idea how easy that would be...

With regard to LTOing DTrace itself... God only knows how many assumptions there are in the codebase that LTO violates. I know the testsuite verifies that we can identify things correctly in the presence of interposition, and that test is obviously not going to work with interposition disabled (but that's not this error, and that's easy to fix in the makefiles by turning on -fno-semantic-interposition for that specific test).

The drti.c failure probably relates to this line:

    dof_hdr_t *dof = &__SUNW_dof;

which suggests that the entire .SUNW_dof section has been dropped as unused. While this is technically true (it isn't used by the program, no), it is distinctly unhelpful. I wonder how we can stop it. We can hardly require everyone linking DOF into their program to use a specialized linker script: I wonder if this requires upstream binutils changes to exempt this section from pruning?

[thesamesam]

This got a little bit better with bb8fa79 and 0fd9028 though I still got:

/usr/lib/gcc/x86_64-pc-linux-gnu/15/../../../../x86_64-pc-linux-gnu/bin/ld: /var/tmp/portage/dev-debug/dtrace-2.0.2/temp/ccQM1qBx.ltrans0.ltrans.o: in function `main':
/var/tmp/portage/dev-debug/dtrace-2.0.2/work/dtrace-utils-2.0.2/test/triggers/usdt-tst-argmap.c:14:(.text.startup+0x1b): undefined reference to `__dtrace_test_prov___place'
/usr/lib/gcc/x86_64-pc-linux-gnu/15/../../../../x86_64-pc-linux-gnu/bin/ld: /var/tmp/portage/dev-debug/dtrace-2.0.2/work/dtrace-utils-2.0.2/test/triggers/usdt-tst-argmap.c:15:(.text.star
tup+0x2c): undefined reference to `__dtrace_test_prov___place2'
/usr/lib/gcc/x86_64-pc-linux-gnu/15/../../../../x86_64-pc-linux-gnu/bin/ld: /var/tmp/portage/dev-debug/dtrace-2.0.2/work/dtrace-utils-2.0.2/test/triggers/usdt-tst-argmap.c:16:(.text.star
tup+0x3d): undefined reference to `__dtrace_test_prov___place3'
/usr/lib/gcc/x86_64-pc-linux-gnu/15/../../../../x86_64-pc-linux-gnu/bin/ld: /var/tmp/portage/dev-debug/dtrace-2.0.2/work/dtrace-utils-2.0.2/test/triggers/usdt-tst-argmap.c:17:(.text.star
tup+0x4e): undefined reference to `__dtrace_test_prov___place4'
collect2: error: ld returned 1 exit status

which I haven't investigated, just nothing it here to link the 2 commits.

[thesamesam]

With e8e9244 (overall, the series starting at https://lore.kernel.org/dtrace/IA4PR10MB83749DB4A1B2117E28EC16A2C269A@IA4PR10MB8374.namprd10.prod.outlook.com/T/#u), it builds fine with LTO \o/

Thank you!

[thesamesam]

.. though I maybe spoke too soon, as it hangs at runtime (no probes fire). I'll file another bug for that when I get a chance.

[kvanhees]

Did you test after install or just from the build? This series changes the dtprobed so it needs to be running the new one for this to work. There is no way to be backwards compatible where new dtrace can generate USDT data that the old dtprobed can handle - only the other way around (previously compiled executables and libraries with USDT probes will work with the new dtprobed).