Missing symbols in libcoreclr.so on SunOS

[am11]

After the CoreCLR debug build, the following symbols are shown missing:

~/runtime$ ldd -r artifacts/bin/coreclr/$(uname).x64.Debug/libcoreclr.so
        libpthread.so.1 =>       /lib/64/libpthread.so.1
        libstdc++.so.6 =>        /opt/local/gcc7/lib/amd64/libstdc++.so.6
        libm.so.2 =>     /lib/64/libm.so.2
        librt.so.1 =>    /lib/64/librt.so.1
        libssp.so.0 =>   /opt/local/gcc7/lib/amd64/libssp.so.0
        libgcc_s.so.1 =>         /opt/local/gcc7/lib/amd64/libgcc_s.so.1
        libc.so.1 =>     /lib/64/libc.so.1
        symbol not found: _ZN9IpcStreamD1Ev             (artifacts/bin/coreclr/SunOS.x64.Debug/libcoreclr.so)
        symbol not found: _ZN9IpcStream5WriteEPKvjRji           (artifacts/bin/coreclr/SunOS.x64.Debug/libcoreclr.so)
        symbol not found: _ZN9IpcStream4ReadEPvjRji             (artifacts/bin/coreclr/SunOS.x64.Debug/libcoreclr.so)
        symbol not found: _ZNK9IpcStream5FlushEv                (artifacts/bin/coreclr/SunOS.x64.Debug/libcoreclr.so)
        symbol not found: _ZN9IpcStream14DiagnosticsIpc7ConnectEPFvPKcjE                (artifacts/bin/coreclr/SunOS.x64.Debug/libcoreclr.so)
        symbol not found: _ZN9IpcStream14DiagnosticsIpc6AcceptEPFvPKcjE         (artifacts/bin/coreclr/SunOS.x64.Debug/libcoreclr.so)
        symbol not found: _ZN9IpcStream14DiagnosticsIpc6CreateEPKcNS0_14ConnectionModeEPFvS2_jE         (artifacts/bin/coreclr/SunOS.x64.Debug/libcoreclr.so)
        symbol not found: _ZN9IpcStream14DiagnosticsIpc6ListenEPFvPKcjE         (artifacts/bin/coreclr/SunOS.x64.Debug/libcoreclr.so)
        symbol not found: _ZN9IpcStream14DiagnosticsIpcD1Ev             (artifacts/bin/coreclr/SunOS.x64.Debug/libcoreclr.so)
        symbol not found: _ZN9IpcStream14DiagnosticsIpc4PollEPNS0_13IpcPollHandleEjiPFvPKcjE            (artifacts/bin/coreclr/SunOS.x64.Debug/libcoreclr.so)
        symbol not found: _ZN9IpcStream14DiagnosticsIpc5CloseEbPFvPKcjE         (artifacts/bin/coreclr/SunOS.x64.Debug/libcoreclr.so)
        symbol not found: _ZN9IpcStream5CloseEPFvPKcjE          (artifacts/bin/coreclr/SunOS.x64.Debug/libcoreclr.so)
        symbol not found: _Z23RtlVirtualUnwind_UnsafejmmP17_RUNTIME_FUNCTIONP8_CONTEXTPPvPmP30_KNONVOLATILE_CONTEXT_POINTERS            (artifacts/bin/coreclr/SunOS.x64.Debug/libcoreclr.so)

with nm(1), I can see, for example, _ZN9IpcStreamD1Ev symbol is in the *.o and *.a on SmartOS when compared with Ubuntu debug x64 build but missing in *.so on SmartOS:

SYMBOL=_ZN9IpcStreamD1Ev
for file in $(find artifacts/ -name *.so -o -name *.o -o -name *.a); do
  echo $file; nm $file 2>/dev/null | grep $SYMBOL
done | grep -B1 $SYMBOL | grep -v '\--'

both are compiled using gcc v7.5.0:

src/coreclr/build-runtime.sh -skipgenerateversion -nopgooptimize    \
    -cmakeargs -DCLR_CMAKE_BUILD_TESTS=1 -gcc

SmartOS:

artifacts/obj/coreclr/SunOS.x64.Debug/src/vm/libcee_dac.a
                 U _ZN9IpcStreamD1Ev
artifacts/obj/coreclr/SunOS.x64.Debug/src/vm/wks/libcee_wks.a
                 U _ZN9IpcStreamD1Ev
                 U _ZN9IpcStreamD1Ev
                 U _ZN9IpcStreamD1Ev
                 U _ZN9IpcStreamD1Ev
                 U _ZN9IpcStreamD1Ev
                 U _ZN9IpcStreamD1Ev
artifacts/obj/coreclr/SunOS.x64.Debug/src/vm/wks/CMakeFiles/cee_wks.dir/__/profilerdiagnosticprotocolhelper.cpp.o
                 U _ZN9IpcStreamD1Ev
artifacts/obj/coreclr/SunOS.x64.Debug/src/vm/wks/CMakeFiles/cee_wks.dir/__/diagnosticserver.cpp.o
                 U _ZN9IpcStreamD1Ev
artifacts/obj/coreclr/SunOS.x64.Debug/src/vm/wks/CMakeFiles/cee_wks.dir/__/fastserializer.cpp.o
                 U _ZN9IpcStreamD1Ev
artifacts/obj/coreclr/SunOS.x64.Debug/src/vm/wks/CMakeFiles/cee_wks.dir/__/dumpdiagnosticprotocolhelper.cpp.o
                 U _ZN9IpcStreamD1Ev
artifacts/obj/coreclr/SunOS.x64.Debug/src/vm/wks/CMakeFiles/cee_wks.dir/__/eventpipeprotocolhelper.cpp.o
                 U _ZN9IpcStreamD1Ev
artifacts/obj/coreclr/SunOS.x64.Debug/src/vm/wks/CMakeFiles/cee_wks.dir/__/ipcstreamfactory.cpp.o
                 U _ZN9IpcStreamD1Ev
artifacts/obj/coreclr/SunOS.x64.Debug/src/vm/CMakeFiles/cee_dac.dir/dumpdiagnosticprotocolhelper.cpp.o
                 U _ZN9IpcStreamD1Ev
artifacts/obj/coreclr/SunOS.x64.Debug/src/debug/debug-pal/CMakeFiles/debug-pal.dir/unix/diagnosticsipc.cpp.o
000000000000111a T _ZN9IpcStreamD1Ev
artifacts/obj/coreclr/SunOS.x64.Debug/src/debug/debug-pal/libdebug-pal.a
000000000000111a T _ZN9IpcStreamD1Ev
artifacts/obj/coreclr/SunOS.x64.Debug/src/dlls/mscoree/coreclr/libcoreclr.so
                 U _ZN9IpcStreamD1Ev
artifacts/bin/coreclr/SunOS.x64.Debug/libcoreclr.so
                 U _ZN9IpcStreamD1Ev
artifacts/bin/coreclr/SunOS.x64.Debug/sharedFramework/libcoreclr.so
                 U _ZN9IpcStreamD1Ev

Ubuntu:

artifacts/obj/coreclr/Linux.x64.Debug/src/vm/libcee_dac.a
                 U _ZN9IpcStreamD1Ev
artifacts/obj/coreclr/Linux.x64.Debug/src/vm/wks/libcee_wks.a
                 U _ZN9IpcStreamD1Ev
                 U _ZN9IpcStreamD1Ev
                 U _ZN9IpcStreamD1Ev
                 U _ZN9IpcStreamD1Ev
                 U _ZN9IpcStreamD1Ev
                 U _ZN9IpcStreamD1Ev
artifacts/obj/coreclr/Linux.x64.Debug/src/vm/wks/CMakeFiles/cee_wks.dir/__/profilerdiagnosticprotocolhelper.cpp.o
                 U _ZN9IpcStreamD1Ev
artifacts/obj/coreclr/Linux.x64.Debug/src/vm/wks/CMakeFiles/cee_wks.dir/__/dumpdiagnosticprotocolhelper.cpp.o
                 U _ZN9IpcStreamD1Ev
artifacts/obj/coreclr/Linux.x64.Debug/src/vm/wks/CMakeFiles/cee_wks.dir/__/fastserializer.cpp.o
                 U _ZN9IpcStreamD1Ev
artifacts/obj/coreclr/Linux.x64.Debug/src/vm/wks/CMakeFiles/cee_wks.dir/__/diagnosticserver.cpp.o
                 U _ZN9IpcStreamD1Ev
artifacts/obj/coreclr/Linux.x64.Debug/src/vm/wks/CMakeFiles/cee_wks.dir/__/eventpipeprotocolhelper.cpp.o
                 U _ZN9IpcStreamD1Ev
artifacts/obj/coreclr/Linux.x64.Debug/src/vm/wks/CMakeFiles/cee_wks.dir/__/ipcstreamfactory.cpp.o
                 U _ZN9IpcStreamD1Ev
artifacts/obj/coreclr/Linux.x64.Debug/src/vm/CMakeFiles/cee_dac.dir/dumpdiagnosticprotocolhelper.cpp.o
                 U _ZN9IpcStreamD1Ev
artifacts/obj/coreclr/Linux.x64.Debug/src/debug/debug-pal/libdebug-pal.a
0000000000001104 T _ZN9IpcStreamD1Ev
artifacts/obj/coreclr/Linux.x64.Debug/src/debug/debug-pal/CMakeFiles/debug-pal.dir/unix/diagnosticsipc.cpp.o
0000000000001104 T _ZN9IpcStreamD1Ev
artifacts/obj/coreclr/Linux.x64.Debug/src/dlls/mscoree/coreclr/libcoreclr.so
0000000000634c70 t _ZN9IpcStreamD1Ev
artifacts/bin/coreclr/Linux.x64.Debug/sharedFramework/libcoreclr.so
0000000000634c70 t _ZN9IpcStreamD1Ev
artifacts/bin/coreclr/Linux.x64.Debug/libcoreclr.so
0000000000634c70 t _ZN9IpcStreamD1Ev

Is there a way to troubleshoot rootcause of these missing IpcStream* and RtlVirtualUnwind_Unsafe symbols from shared object?

cc @janvorli

[am11]

Here is how the linker invocation looks like:

cat artifacts/obj/coreclr/$(uname).x64.Debug/src/dlls/mscoree/coreclr/CMakeFiles/coreclr.dir/link.txt

SmartOS

/opt/local/bin/g++ -fPIC -fstack-protector -g -Wl,-M,/home/am11/runtime/artifacts/obj/coreclr/SunOS.x64.Debug/src/dlls/mscoree/coreclr/coreclr.exports -shared -Wl,-hlibcoreclr.so -o libcoreclr.so CMakeFiles/coreclr.dir//mscoree.cpp.o CMakeFiles/coreclr.dir//unixinterface.cpp.o CMakeFiles/coreclr.dir/////version.c.o ../../../utilcode/libutilcode.a ../../../debug/ee/wks/libcordbee_wks.a ../../../debug/debug-pal/libdebug-pal.a ../../../unwinder/libunwinder_wks.a ../../../vm/wks/libcee_wks.a ../../../binder/libv3binder.a ../../../md/compiler/libmdcompiler_wks.a ../../../md/runtime/libmdruntime_wks.a ../../../md/enc/libmdruntimerw_wks.a ../../../md/hotdata/libmdhotdata_full.a ../../../classlibnative/bcltype/libbcltype.a ../../../md/ceefilegen/libceefgen.a ../../../classlibnative/float/libcomfloat_wks.a ../../../inc/libcorguids.a ../../../gcinfo/libgcinfo.a ../../../debug/ildbsymlib/libildbsymlib.a ../../../utilcode/libutilcode.a ../../../binder/libv3binder.a ../../../libraries-native/liblibraries-native.a ../../../libraries-native/System.Globalization.Native/libSystem.Globalization.Native.a ../../../interop/libinterop.a ../../../pal/src/libcoreclrpal.a ../../mscorrc/libmscorrc.a ../../../palrt/libpalrt.a ../../../vm/eventing/eventpipe/libeventpipe.a ../../../pal/src/eventprovider/dummyprovider/libeventprovider.a ../../../nativeresources/libnativeresourcestring.a -lpthread -lrt

Ubuntu

/usr/bin/g++-7 -fPIC -g -Wl,--version-script=/home/vagrant/runtime/artifacts/obj/coreclr/Linux.x64.Debug/src/dlls/mscoree/coreclr/coreclr.exports -Wl,--build-id=sha1 -Wl,-z,relro,-z,now -Wl,-Bsymbolic -shared -Wl,-soname,libcoreclr.so -o libcoreclr.so CMakeFiles/coreclr.dir//mscoree.cpp.o CMakeFiles/coreclr.dir//unixinterface.cpp.o CMakeFiles/coreclr.dir/////version.c.o ../../../utilcode/libutilcode.a -Wl,--start-group ../../../debug/ee/wks/libcordbee_wks.a ../../../debug/debug-pal/libdebug-pal.a ../../../unwinder/libunwinder_wks.a ../../../vm/wks/libcee_wks.a ../../../binder/libv3binder.a -Wl,--end-group ../../../md/compiler/libmdcompiler_wks.a ../../../md/runtime/libmdruntime_wks.a ../../../md/enc/libmdruntimerw_wks.a ../../../md/hotdata/libmdhotdata_full.a ../../../classlibnative/bcltype/libbcltype.a ../../../md/ceefilegen/libceefgen.a ../../../classlibnative/float/libcomfloat_wks.a ../../../inc/libcorguids.a ../../../gcinfo/libgcinfo.a ../../../debug/ildbsymlib/libildbsymlib.a ../../../utilcode/libutilcode.a ../../../binder/libv3binder.a ../../../libraries-native/liblibraries-native.a ../../../libraries-native/System.Globalization.Native/libSystem.Globalization.Native.a ../../../interop/libinterop.a -Wl,--whole-archive ../../../pal/src/libcoreclrpal.a -Wl,--no-whole-archive ../../mscorrc/libmscorrc.a ../../../palrt/libpalrt.a -Wl,--whole-archive ../../../pal/src/libtracepointprovider.a -Wl,--no-whole-archive ../../../vm/eventing/eventpipe/libeventpipe.a ../../../pal/src/eventprovider/lttngprovider/libeventprovider.a ../../../nativeresources/libnativeresourcestring.a -lgcc_s -lpthread -lrt -ldl

[jclulow]

@am11 It doesn't seem like diagnosticsipc.cpp.o (where it seems like _ZN9IpcStreamD1Ev is defined) appears in the link line directly there. Do you know if it is included indirectly, by being part of one of the archives (*.a) files that appear there? (Which, and how was that built?)

[am11]

Updated above comments with results from gcc 7.5.0 on Ubuntu as well.

@jclulow, it is linked in debug-pal:

SmartOS:

$ cat artifacts/obj/coreclr/$(uname).x64.Debug/src/debug/debug-pal/CMakeFiles/debug-pal.dir/link.txt

/opt/local/bin/ar qc libdebug-pal.a  CMakeFiles/debug-pal.dir/unix/diagnosticsipc.cpp.o CMakeFiles/debug-pal.dir/unix/twowaypipe.cpp.o CMakeFiles/debug-pal.dir/unix/processdescriptor.cpp.o CMakeFiles/debug-pal.dir/__/__/__/version.c.o
/opt/local/bin/ranlib libdebug-pal.a

Ubuntu:

/usr/bin/ar qc libdebug-pal.a  CMakeFiles/debug-pal.dir/unix/diagnosticsipc.cpp.o CMakeFiles/debug-pal.dir/unix/twowaypipe.cpp.o CMakeFiles/debug-pal.dir/unix/processdescriptor.cpp.o CMakeFiles/debug-pal.dir/__/__/__/version.c.o
/usr/bin/ranlib libdebug-pal.a

[am11]

Related: #36426 - ubuntu -gcc build is also missing some different symbols. So some investigation is needed for gcc support in general.

[janvorli]

@am11 can you try to add the debug-pal one more time to the list of libraries in CORECLR_LIBRARIES, at its end? It seems as if the START_LIBRARY_GROUP / END_LIBRARY_GROUP was somehow not working properly.

[am11]

@janvorli, thank you for the tip. START_LIBRARY_GROUP and friends were missing for SmartOS in some cmake scripts. I have adjusted those scripts and added some linkages (-lsocket, -lm and -lz in few places).