Can not load or initialize libmscordaccor.so when debugging a coredump in lldb

[klesta490]

Hi,
I am trying to investigate our application crash

• net core app running in docker
• docker based on image microsoft/dotnet:2.1.7-runtime-stretch-slim
• there is no .net runtime on host, only in docker image
• host is centos

Our app is crashing with
kernel: traps: dotnet[12953] general protection error:0 in libgcc_s.so.1[7fbb0da04000+16000] error message in system logs (propably some native lib we call from C# is causing memory corruption).
I decided co create dump on next crash. So I setup COMPlus_DbgEnableMiniDump=1 and COMPlus_DbgMiniDumpType=4 (because I want to see method names #56). Also I run docker container as --privileged so dump is possible. When error occurs, dump is successfully created in /tmp directory.
I create image from crashed container so I can inspect dump
docker commit -m "crash" <container-name>.

Now I run this container with /bin/bash entrypoint and continue analysis with lldb-4.0 (as far as I know 3.9 is minimal working version).

I open dump using
lldb-4.0 $(which dotnet) --core /tmp/coredump.1
lldb> plugin load /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.7/libsosplugin.so
lldb> setclrpath /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.7
now I run any sos command and I get

Failed to load data access DLL, 0x80004005 Can not load or initialize libmscordaccore.so. The target runtime may not be initialized.

Really weird is, that this approach is working flawlessly with demo project I have written to check this dump analysis scenario.

I have found #42 but I don't think that any advice from this issue is relevant to me, because I analyze dump with exactly same environment with which dump was created.

Thanks for help, any advice will be appreciated

[patricksuo]

You could try my docker image su21/dotnet:2.2-sdk-dbg-1

highlights:

• backported some bugfix/enhancements to 2.2: dotnet/coreclr@release/2.2...sillyousu:sims2.2
• bundle some useful stuff like lldb3.9, runtime/BCL symbols, runtime indep sosplugin

/usr/share/dotnet/shared/Microsoft.NETCore.App/2.2.0/createdump $PID

# and then

/diagnostics/opencore $CORE_FILE_PATH

[mikem8361]

What does “target modules list” display? Are all the modules listed or just “dotnet” host? If the core dump is loaded in exactly the same environment as it was taken, then “setclrpath” should not be necessary. This error usually means that it can’t find the DAC or it isn’t the right version. The new SOS in the diagnostics repo https://github.com/dotnet/diagnostics.git may make this easier. See the README.md on building it. mikem

[mikem8361]

These instructions might help a little: https://github.com/dotnet/diagnostics/blob/master/documentation/debugging-coredump.md.

It uses the dotnet-symbol download tool to get all the proper bits (especially "dotnet") and symbols, but that may be moot since your environment should be the same as when the core dump was generated.

[klesta490]

@sillyousu I'll give it a try, thx. But as far as I understand I need to rebuild my app with this as base image? I suppose otherwise binaries wouldn't match?

[patricksuo]

If you are using Framework-dependent deployment , I think you don’t have to.

https://docs.microsoft.com/en-us/dotnet/core/deploying/

[klesta490]

@mikem8361

(lldb) target module list
[  0] BC60FC80-BA5B-F4B2-8EF9-D82325D2D494-EDBDAFE6                    /usr/bin/dotnet

but I have also noticed
that I got more files in /tmp

-rw-------. 1 root root    22409216 Feb 12 15:22 core.a7aec291fc82.createdump.1549984934.504
-rw-------. 1 root root    22409216 Feb 12 17:50 core.a7aec291fc82.createdump.1549993849.2021
-rw-------. 1 root root 58385043456 Feb 13 07:37 core.a7aec291fc82.dotnet.1550043456.1
-rw-r--r--. 1 root root 29160243200 Feb 13 07:37 coredump.1

[mikem8361]

That there is only the dotnet module is the problem. This is a lldb quirk. Either the dotnet module doesn't match the one in core dump or the core dump is the one running your app if you ran your app with "dotnet run". That launches the dotnet program which launches your app. If you run dotnet /path/to/app.dll then you should get the right dump.

The "coredump.1" is dump generated by createdump. Processes in docker containers usually start with pid 1. The above coredumps with "createdump" in them means that createdump must have faulted with is an issue that was fixed in 2.1.8 (I think) and 2.2.1 (and master/3.0 branch).

If you can install the dotnet-symbol global tool and run it on /tmp/coredump.1 and try loading it again, the right version of dotnet and the rest of the binaries should be available.

[klesta490]

Ok, we reproduces problem again on 2.2.2 runtime, I am still unable to open dump :-/.

root@4696fcf1b0ab:/app# lldb-4.0 $(which dotnet) --core /tmp/coredump.1
(lldb) target create "/usr/bin/dotnet" --core "/tmp/coredump.1"
Core file '/tmp/coredump.1' (x86_64) was loaded.
(lldb) plugin load /usr/share/dotnet/shared/Microsoft.NETCore.App/2.2.2/libsosplugin.so
(lldb) setclrpath /usr/share/dotnet/shared/Microsoft.NETCore.App/2.2.2
Set load path for sos/dac/dbi to '/usr/share/dotnet/shared/Microsoft.NETCore.App/2.2.2/'
(lldb) sos DumpStack
Failed to load data access DLL (libmscordaccore.so), 0x80004005
Some functionality may be impaired
OS Thread Id: 0x1 (1)
TEB information is not available so a stack size of 0xFFFF is assumed
Current frame: 00007ff9612de15f
Child-SP         RetAddr          Caller, Callee
DumpStack  failed

(lldb) target module list
[  0] A0ABA4F8-ABD7-1A84-46F5-FA05F4C4638D-2AC86D60                    /usr/bin/dotnet

About dotnet symbol:

dotnet symbol coredump.1
Downloading from http://msdl.microsoft.com/download/symbols/
ERROR: Unexpected end of data: Expected 1 bytes.

[mikem8361]

Can you zip/attach the coredump.1? There is something very strange going on.

[klesta490]

I'll upload it on monday. Right now here is log from createdump with -d our applications modules are mentioned there (like /app/Users.dll)
log.txt

There are some errors - propably related?

cat log.txt | grep FAIL
ReadMemory(0x7ffd3e006000, 40) ehdr FAILED
Unwind: stack walker GetContext FAILED
Unwind: stack walker GetContext FAILED
Unwind: stack walker GetContext FAILED
Unwind: stack walker GetContext FAILED
Unwind: stack walker GetContext FAILED
Unwind: stack walker GetContext FAILED
Unwind: stack walker GetContext FAILED
Unwind: stack walker GetContext FAILED
Unwind: stack walker GetContext FAILED
Unwind: stack walker GetContext FAILED
Unwind: stack walker GetContext FAILED
Unwind: stack walker GetContext FAILED
Unwind: stack walker GetContext FAILED
Unwind: stack walker GetContext FAILED

but target module list shows just dotnet

[klesta490]

Ok, I got a repro. Createdump does not work when ChakraCore is loaded into project. (By does not work I mean dump looks like it's corrupted)

<ItemGroup>
    <PackageReference Include="JavaScriptEngineSwitcher.ChakraCore" Version="3.0.6" />
    <PackageReference Include="JavaScriptEngineSwitcher.ChakraCore.Native.linux-x64" Version="3.0.6" />
    <PackageReference Include="JavaScriptEngineSwitcher.Core" Version="3.0.0" />
  </ItemGroup>

static void Main(string[] args)
{
       var engine = new JavaScriptEngineSwitcher.ChakraCore.ChakraCoreJsEngineFactory().CreateEngine();
       Console.ReadLine();
}

Once https://github.com/Microsoft/ChakraCore engine is created, process dumps are corrupted.

Maybe it is somehow related to signal handling? We have also found a bug that SIGSEGV handler in ChakraCore is buggy (stack smashing).