Linux arm 5.0 launch fails with CORDBG_E_MISSING_DEBUGGER_EXPORTS

[gregg-miskelly]

This issue is from dotnet/vscode-csharp#4210. Bug report is from @profix898

Issue Description

After deploying a .NET 5 linux-arm app to a Raspberry Pi 4, debugging fails with CORDBG_E_MISSING_DEBUGGER_EXPORTS (0x80131c4f). This doesn't happen with 3.1.

Steps to Reproduce

1. I have created a minimal console project with the following content:

RPiDemo.csproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net5.0</TargetFramework>
  </PropertyGroup>
</Project>

Program.cs

using System;

namespace RPiDemo
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Hello RPi!");
        }
    }
}

The project is successfully compiled in VS Code on Win 10 x64 using the locally installed .NET 5 SDK. The project is published for 'linux-arm' and copied to the Raspberry Pi 4 (with Raspian 10, latest version as of today) with the following command:

dotnet publish -c Release -r linux-arm -o ./publish/linux-arm ${workspaceFolder} ; scp -rp -P 2222 ./publish/linux-arm/ root@192.168.81.129:/opt/RPiDemo

I'm essentially following the instructions at https://github.com/OmniSharp/omnisharp-vscode/wiki/Remote-Debugging-On-Linux-Arm (or similar tutorials on the internet).

The project is then launched on the RPi with the 'vsdbg' remote debugger. My launch.json looks like this:

{
    "configurations": [        
        {
            "name": "Launch LOCAL",
            "type": "coreclr",
            "request": "launch",
            "preLaunchTask": "build",
            "program": "${workspaceFolder}/bin/Debug/net5.0/RPiDemo.dll",
            "args": [],
            "cwd": "${workspaceFolder}",
            "stopAtEntry": false,
            "console": "internalConsole"
        }, 
        {
            "name": "Launch REMOTE",
            "type": "coreclr",
            "request": "launch",
            "preLaunchTask": "BuildPublishCopyLinuxARM",
            "program": "/usr/bin/dotnet",
            "args": [
                "/opt/RPiDemo/RPiDemo.dll"
            ],
            "cwd": "/opt/RPiDemo",
            "stopAtEntry": false,
            "console": "internalConsole",
            "pipeTransport": {
                "pipeCwd": "${workspaceRoot}",
                "pipeProgram": "ssh",
                "pipeArgs": [
                    "-T",
                    "-p", "2222",
                    "root@192.168.81.129"
                ],
                "quoteArgs": true,
                "debuggerPath": "/usr/bin/vsdbg/vsdbg"
            },
            "logging": {
                "engineLogging": true,
                "programOutput": true,
                "exceptions": true
            }
        },
    ]
}

Expected Behavior

Debugger should launch and attach to the process on the RPi and provide debug information to VS Code.

Actual Behavior

Debugger prints the following error message: Unable to attach to CoreCLR. Unknown Error: 0x80131c4f.
I have then enabled logging via "engineLogging": true, to obtain more details (as shown in the log below).

I'd like to note, that the console project works perfectly both locally on the Win 10 x64 host and also when being launched manually on the RPi via the dotnet RPiDemo.dll command (directly in the published directory '/opt/RPiDemo').

[ghost]

Tagging subscribers to this area: @tommcdon
See info in area-owners.md if you want to be subscribed.

Issue Details

Description:	This issue is from dotnet/vscode-csharp#4210. Bug report is from @profix898 Issue Description After deploying a .NET 5 linux-arm app to a Raspberry Pi 4, debugging fails with CORDBG_E_MISSING_DEBUGGER_EXPORTS (0x80131c4f). This doesn't happen with 3.1. Steps to Reproduce I have created a minimal console project with the following content: RPiDemo.csproj <Project Sdk="Microsoft.NET.Sdk"> <PropertyGroup> <OutputType>Exe</OutputType> <TargetFramework>net5.0</TargetFramework> </PropertyGroup> </Project> Program.cs using System; namespace RPiDemo { class Program { static void Main(string[] args) { Console.WriteLine("Hello RPi!"); } } } The project is successfully compiled in VS Code on Win 10 x64 using the locally installed .NET 5 SDK. The project is published for 'linux-arm' and copied to the Raspberry Pi 4 (with Raspian 10, latest version as of today) with the following command: dotnet publish -c Release -r linux-arm -o ./publish/linux-arm ${workspaceFolder} ; scp -rp -P 2222 ./publish/linux-arm/ root@192.168.81.129:/opt/RPiDemo I'm essentially following the instructions at https://github.com/OmniSharp/omnisharp-vscode/wiki/Remote-Debugging-On-Linux-Arm (or similar tutorials on the internet). The project is then launched on the RPi with the 'vsdbg' remote debugger. My launch.json looks like this: { "configurations": [ { "name": "Launch LOCAL", "type": "coreclr", "request": "launch", "preLaunchTask": "build", "program": "${workspaceFolder}/bin/Debug/net5.0/RPiDemo.dll", "args": [], "cwd": "${workspaceFolder}", "stopAtEntry": false, "console": "internalConsole" }, { "name": "Launch REMOTE", "type": "coreclr", "request": "launch", "preLaunchTask": "BuildPublishCopyLinuxARM", "program": "/usr/bin/dotnet", "args": [ "/opt/RPiDemo/RPiDemo.dll" ], "cwd": "/opt/RPiDemo", "stopAtEntry": false, "console": "internalConsole", "pipeTransport": { "pipeCwd": "${workspaceRoot}", "pipeProgram": "ssh", "pipeArgs": [ "-T", "-p", "2222", "root@192.168.81.129" ], "quoteArgs": true, "debuggerPath": "/usr/bin/vsdbg/vsdbg" }, "logging": { "engineLogging": true, "programOutput": true, "exceptions": true } }, ] } Expected Behavior Debugger should launch and attach to the process on the RPi and provide debug information to VS Code. Actual Behavior Debugger prints the following error message: Unable to attach to CoreCLR. Unknown Error: 0x80131c4f. I have then enabled logging via "engineLogging": true, to obtain more details (as shown in the log below). I'd like to note, that the console project works perfectly both locally on the Win 10 x64 host and also when being launched manually on the RPi via the dotnet RPiDemo.dll command (directly in the published directory '/opt/RPiDemo').
Author:	gregg-miskelly
Assignees:	-
Labels:	area-Diagnostics-coreclr, untriaged
Milestone:	-

[tommcdon]

@hoyosjs

[krwq]

should this be considered for servicing fix? I'm hearing users not wanting to move to 5.0 because of this

[Ellerbach]

+1, would be great to see it fixed, debugging is one of the key value of .NET on devices like Raspberry Pi running Linux arm based OS.

[hoyosjs]

should this be considered for servicing fix? I'm hearing users not wanting to move to 5.0 because of this

@krwq Do you know if this is just arm linux customers? I was looking at it this past week, but had to prioritize other work. I'll continue this week then.

[profix898]

From my experience, it is just Linux ARM. Even Linux ARM64 works fine.
I switched to the beta version of Raspian 64-bit (and tried Ubuntu 64-bit for Raspberry Pi), on both platforms the debugger works as expected. Note, however, that the 32-bit version is the default/stable branch for RPi.

[wsad4ryba]

I also encountered this problem. In fact, it's a bit annoying because after the update it turns out that the best solution is to stay with the old version.

[janvorli]

@hoyosjs just a quick idea. Since it is ARM only, I wonder if it could be yet another case of the address conversion issue that we've hit in the past. I can see that all cases that can fail with that error call GetDacTableAddress that ends up calling TryGetSymbol in the elfreader.cpp with m_globalBase passed as the base address parameter. That variable is set here:

runtime/src/coreclr/src/debug/daccess/daccess.cpp

Line 5594 in e8c4f22

m_globalBase = TO_TADDR(base);

I wonder if the problem could be caused by the TO_TADDR being used to do that conversion.

[janvorli]

Or the base being already incorrectly sign extended.

[hoyosjs]

I haven't forgotten this. I just can't seem to repro this. All my remote attaches have been working on a RPi running Raspbian 10.

[jefflill]

We're seeing the same issue with our Visual Studio Raspberry Debugger extension. We're seeing the same error: 0x80131c4f

It should be pretty quick for you to install and configure the extension because it handles installing the .NET SDK and vsdbg on the Raspberry and also configures the SSH key pair for you. Here's our repo and getting started instructions:

https://github.com/nforgeio/RaspberryDebugger

It would be really nice to get this working.

[hoyosjs]

I just tested this again and I am able to attach on 5.0 with no issues on both a RPi 2 and RPi 3 running raspbian 10. However, given this many reports of things failing, there's definitely something that's not working. I copied the programs from the original issue. The files I used in VS Code for the attach are:

launch.json

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Launch REMOTE",
            "type": "coreclr",
            "request": "launch",
            "preLaunchTask": "BuildPublishCopyLinuxARM",
            "program": "/home/pi/juhoyosa/opt/RPiDemo/RPiDemo",
            "args": [],
            "cwd": "/home/pi/juhoyosa/opt/RPiDemo",
            "stopAtEntry": true,
            "console": "internalConsole",
            "justMyCode": false,
            "pipeTransport": {
                "pipeCwd": "${workspaceRoot}",
                "pipeProgram": "ssh.exe",
                "pipeArgs": [
                    "-T",
                    "pi@diag-pi3"
                ],
                "quoteArgs": true,
                "debuggerPath": "/home/pi/vsdbg/vsdbg"
            },
            "logging": {
                "engineLogging": true,
                "programOutput": true,
                "exceptions": true
            }
        },
    ]
}

tasks.json

{
    // See https://go.microsoft.com/fwlink/?LinkId=733558
    // for the documentation about the tasks.json format
    "version": "2.0.0",
    "tasks": [
        {
            "label": "PublishForRPi",
            "command": "dotnet",
            "type": "shell",
            "args": [
                "publish",
                "-c", "Release",
                "-r", "linux-arm",
                "-o", "./publish/linux-arm",
                "${workspaceFolder}"
            ],
            "group": "build",
            "problemMatcher": "$msCompile"
        },
        {
            "label": "Deploy",
            "type": "shell",
            "command": "scp.exe",
            "args": [
                "-rp",
                "./publish/linux-arm/",
                "pi@diag-pi3:/home/pi/juhoyosa/opt/RPiDemo",
            ],
            "group": "build",
            "presentation": {
                "reveal": "silent"
            },
            "problemMatcher": "$msCompile"
        },
        {
            "label": "BuildPublishCopyLinuxARM",
            "dependsOrder": "sequence",
            "dependsOn": ["PublishForRPi", "Deploy"]
        }
    ]
}

@gregg-miskelly one thing I did notice is that I have to use justMyCode for stopAtEntry to work. Is this a known issue?

[gregg-miskelly]

You need to disable justMyCode because you used release configuration. Publish debug instead for better results (though no idea if that will help making the problem reproduce).

[gregg-miskelly]

@profix898 looking more closely at your launch.json, are these guesses correct?

1. You are building and running dotnet publish on your VS Code machine
2. You have the ARM .NET Runtime installed on your Pi

Is that correct?
When you publish, do you specify, "-r", "linux-arm", or are you publishing your app as framework-dependent?
Does the problem go away if you try Juan's approach of publishing a self-contained project? (note that you want to swap out the program in your launch.json if you do this)

[hoyosjs]

@gregg-miskelly the original post you shared indicated this was a self-contained publish that gets deployed. I had also tried with a fx dependent one, and it worked too. Is it possible that the deployed debugger bits are stale? Or does this one also auto-update? I know if does update under the VS extension, but I don't know what happens when deployed like this. Even with this, I can't think of a recent change in dbgshim that would cause this.

[janvorli]

@hoyosjs the fact that it works for some people and doesn't work for others might really indicate that it is the address conversion issue I've mentioned above. Maybe that on your device, libcoreclr.so is located at a VA < 0x80000000 while on the failing ones it is >= 0x80000000.

[hoyosjs]

@janvorli i did consider your comments, sorry. At a quick look i didn't see anything particularly obvious. I can't recall if 5.0 is when we made ensured PIE/PIC on Linux arm. If I can't force to load at a higher address later (need to research it there's some way to achieve this), I'll probably instrument the runtime and request for someone to help here.

[janvorli]

I'll give it a try on my local device where I have hit the address related issues in the past.
Also, if you are testing it when running the runtime under lldb or gdb, please note that address space randomization is disabled by default under the debugger and you need to explicitly enable it. For lldb, the command is settings set target.disable-aslr false

[hoyosjs]

I did not check if ASLR is disabled OS-wide, though I'd be surprised. I tested this without a native debugger attached. Thanks for checking this @janvorli

[gregg-miskelly]

@hoyosjs The log that profix898 posted indicates that he is using vsdbg 16.8.11013.1, which is the current shipping version.

[gregg-miskelly]

@hoyosjs profix898's launch.json has "program": "/usr/bin/dotnet" which is a bit odd for a self-contained app, so that is why I was asking to make sure he really was really using a self-contained app.

[janvorli]

I've tried locally to debug a test app on Linux arm32 using lldb + SOS. I would expect the same problem to occur, as it loads mscordaccore and accesses libcoreclr symbols. In my case, all code was at addresses >= 0x80000000 including libcoreclr.so, so it seems to invalidate my theory.
I wonder if it is somehow possible that the debugger tries to load a wrong (outdated) mscordaccore.

[janvorli]

I will try the original repro steps on my Odroid XU4 ARM32 device and see if I can repro the issue.

[hoyosjs]

Sorry, but there's no work around. This is a compile time issue. It's been there for years. The problem is we added a new code path and it hits it very easily on such systems. The change is simple but I've had trouble testing this. I've prepared the PR and the port to 5.0 so I can expedite the process.

[bjorg]

Happy to help test if it's not too involved since I'm running the affected hardware.

[hoyosjs]

This has been merged both into master and 5.0. It will be available in the next release.

[hoyosjs]

Happy to help test if it's not too involved since I'm running the affected hardware.

Thanks @bjorg. I was doing manual testing, I wanted to load some tests I can't share but was hitting some issues with the loader being unable to find the right libdl, and couldn't find a combination of LD_LIBRARY_PATH and LD_DEBUG that worked or gave me enough info as to why things failed to load. I used the manual testing as enough proof for now and will investigate doing a more comprehensive testing pass.

[microhobby]

@hoyosjs is this already in the daily installer? Or is there a daily that we can use to test?

[hoyosjs]

It's not going to be officially released at least until next month. There might be a build which you could use to test things out. How do you build/deploy your app? A shared framework deployment (where the runtime is installed for all apps in the machine)? A self contained app deployment (dotnet publish with a specific runtime identifier)? @microhobby

[microhobby]

For debugging we usually run with the shared framework @hoyosjs

[hoyosjs]

Let me ask if I can share a nightly build of the rolling release and answer tomorrow.

[hoyosjs]

There's this https://dotnetcli.blob.core.windows.net/dotnet/Runtime/5.0.2-servicing.20611.21/dotnet-runtime-5.0.2-linux-arm.tar.gz. You can extract it and you'll have a shared framework. On the debugger side, you can set the environment variable DOTNET_ROOT pointing to that folder and use that dotnet executable to ensure you use it. Given that the extracted folder will be called 5.0.2, be really careful to delete it and actually install the release once it actually comes out. This is not a complete release - it's a nightly build and you should use it with caution, there might be bugs as it hasn't gone through all the exit validation processes. It should at least unblock you on debugging as it has the bits that fixed the issue.

[microhobby]

@hoyosjs ok, got it, thanks

[microhobby]

@hoyosjs I try to debug with the framework that you shared but I'm having the same issue:

-------------------------------------------------------------------
You may only use the Microsoft .NET Core Debugger (vsdbg) with
Visual Studio Code, Visual Studio or Visual Studio for Mac software
to help you develop and test your applications.
-------------------------------------------------------------------
Unable to attach to CoreCLR. Unknown Error: 0x80131c4f

Return from dotnet --info

Host (useful for support):
  Version: 5.0.2
  Commit:  988fa3b75f

.NET SDKs installed:
  No SDKs were found.

.NET runtimes installed:
  Microsoft.NETCore.App 5.0.2 [/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET runtimes or SDKs:
  https://aka.ms/dotnet-download

[hoyosjs]

@microhobby let me test that particular build.

[microhobby]

@hoyosjs ok, a quick look on the commits and the 131a30a is there before the 988fa3b 🤷‍♂️

[hoyosjs]

hmm @microhobby I just tried attaching to my repro with the provided sample and I am able to attach using a shared framework. If you add this to your csproj (I assume it's a console-like app), does it still run?

  <PropertyGroup> 
    <RuntimeFrameworkVersion>5.0.1</RuntimeFrameworkVersion>
  </PropertyGroup> 

[microhobby]

@hoyosjs I added the snippet on the .csproj and yes it still runs, with the same behavior
I got your point, I'm running the .dll but checking my settings I'm publishing in self contained.
sorry for confusing you

[microhobby]

@hoyosjs it's working!

I put the --no-self-contained on my build command and now it's using the shared framework.
Thanks!