Can not load or initialize libmscordaccor.so when debugging a coredump in lldb

[JoeStead]

We have a coredump from our application that was built using the .NET Core 2.0.3 SDK, a standalone app. The application was running on CentOS7.

To perform some analysis on the dump, I've exported the file onto an Ubuntu 16.04 VM, with lldb-3.6 installed, and the same .NET Core SDK Version that was used to build the application.

I'm launching lldb like this:

lldb-3.6 /usr/bin/dotnet --core ./mydumpfilehere

The dump is loaded, and I'm able to use the standard lldb commands, so far, so good. Now I want to load the sos plugin, which I do by running:

plugin load /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libsosplugin.so

Now, if I type help, I see the commands from the plugin.

Finally, I setclrpath to setclrpath /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3 and to my knowledge, I should be good to go.

Now, as soon as I use any sos command, I get the following message:

Failed to load data access DLL, 0x80004005
Can not load or initialize libmscordaccore.so. The target runtime may not be initialized.

I thought, that perhaps, the SDK versions weren't quite matching up, so I built the coreclr repo, and used the plugin from there instead, still with no luck.

Finally, I thought I might be able to use the plugin that's actually shipped with our self contained app, again, that does not work.

The only thing I can think of, is I need to analyse this on a CentOS 7 machine, but my (limited) understanding is that shouldn't matter?

[JoeStead]

Thought I'd give 2.1 a go too. Downloaded the 2.1 preview, built a standalone app for Centos, which just printed hello world and threw an exception. Used (still on Ubuntu 16.04) lldb-3.9 to load the dump, and load the plugin from the 2.1 install, exactly the same issue.

[RussKeldorph]

@dotnet/dotnet-diag

[mikem8361]

I think the problem might be creating the dump on Centos but using the Ubuntu runtime/DAC to load it even though it should be the same runtime unless it is a slight different build. It does look like you are using 2.0.3 on both Centos and Ubuntu. I'll try creating a dump on Centos and loading it on my Ubuntu 14.04 machine to see what might be the problem.

Another problem you will run into is using lldb 3.6 on to load the core dump. SOS doesn't work on lldb 3.6 on core dumps. You need 3.9. The problem is that the libsosplugin.so is built for 3.6 on Centos/Ubuntu for 2.0.x. Attached is a 3.9 libsosplugin.so if you want to try that.

For .NET Core 2.1 libsosplugin.so is built for lldb 3.9.

libsosplugin.zip

[JoeStead]

Thanks for the plugin

If there is a problem with creating a dump on Centos and loading on Ubuntu, am I right in thinking I could compile lldb on Centos and hopefully everything will work? Something I've been keen to avoid because it takes so long to build, but needs must :)

[mikem8361]

I have repro'ed the problem of creating a core dump on Centos 7 and attempting to load it on Ubuntu (14.04). I'm currently investigating.

[mikem8361]

I found a simple workaround for now: copy /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclr.so to the directory containing the core dump which I assume is the current directory. That seems to fix the problem. I'll continue to investigate an actual fix.

[mikem8361]

The libcoreclr.so needs to be in the same directory as the host ("dotnet" or "corerun") used to run the .NET Core program used to create the dump. When you load a core dump lldb needs the host program as part of the --core command line.

lldb-3.9 --core coredumpfile dotnet

[JoeStead]

I've not been able to use the workaround you've suggested, just so I'm not misunderstanding the steps:

Use lldb-3.9 (with the 2.0.3 SDK)
Use the version of the plugin you included above
Copy the libcoreclr.so to the same directory as the coredump file
Load the plugin included above
set the clr path?

I'm getting a little confused with the latter steps I think, and it's definitely my fault, this is a whole new world for me

[mikem8361]

You have correct steps (yes on setclrpath). The only thing that seems to be missing in your above comments is the host ("dotnet" or "corerun") used to run your test program. lldb doesn't seem to find any of the other modules if I don't add the host program to the --core comand line.

Here is the script I use to load my core dump on Ubuntu:

BUILD_DIR=$HOME/coreclr/bin/Product/Linux.x64.Debug
DOTNET=/usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3
HOST=$HOME/save/centosdump/dotnet
$LLDB_PATH -o "plugin load $BUILD_DIR/libsosplugin.so" -o "setclrpath $DOTNET" --core $HOME/save/centosdump/core.43690 $HOST

If you only have 2.0.3 installed then the HOST path should be HOST=/usr/share/dotnet/dotnet

$ ls
core.43690*  dotnet*  loadcore*

Hopefully that is clearer.

[mikem8361]

I investigated the problem some more and it is a lldb problem than a plugin/SOS. The lldb APIs that SOS uses to initialize the DAC do not work unless the actual "libcoreclr.so" file matching the one in core dump can be loaded from lldb. Hence requiring libcoreclr.so to be in the directory where the "host" is which is the way lldb finds and loads the module/symbol info for all the modules in the core dump. I've tried all the different symbol and module related settings with no luck. There seems to be no programmatic way to fix this either, but I'll continue to look at this.

[raffaeler]

I tried with the 2.1.300 on Ubuntu18 + lldb-3.9 and the problem is still there.
The easiest workaround is to work with a self contained deployment.
But even in this case, certain commands such as clrstack make lldb crash randomly and you have to restart the debugging from the beginning.

[mikem8361]

@raffaeler Can you give me some more details? Are you debugging a core dump? Do you have any details on how commands like clrstack make lldb crash randomly (repro steps, etc.)?

I'm currently working on addressing problems that make SOS/lldb unreliable or problematic by increasing the test coverage, etc. in the new diagnostics repo.

[raffaeler]

It is a console app that throws right after pressing a key from the console.
Instead of throwing manually I iterate a string characters longer than its length
Lldb catches the exception and I can use sos commands only because I published a self contained deployment.
Commands like pe works, bit clrstack made lldb crash.
Ubuntu is version 18.04 LTS.

[raffaeler]

BTW I attach the process instead of running it through corerun.