dotnet tool install doesn't understand multi-arch systems or TFM requirements

[mhutch]

A user of the .NET global tool vsmac-cli reported the following host error when trying to use the tool after installing it.

% vsmac
A fatal error occurred. The required library libhostfxr.dylib could not be found.
If this is a self-contained application, that library should exist in [/Users/[USER]/.dotnet/tools/.store/vsmac-cli/0.2.0/vsmac-cli/0.2.0/tools/net5.0/any/].
If this is a framework-dependent application, install the runtime in the global location [/usr/local/share/dotnet/x64] or use the DOTNET_ROOT environment variable to specify the runtime location or register the runtime location in [/etc/dotnet].

The .NET runtime can be found at:
- https://aka.ms/dotnet-core-applaunch?missing_runtime=true&arch=x64&rid=osx.12-x64&apphost_version=6.0.3

This error occurred because the app is framework dependent and the end user does not have the required framework installed. This is likely a common scenario for end users, yet the message is needlessly complex, with implementation details (libhostfxr) and irrelevant information (self-contained vs shared) that appears to be targeted at the developer of the app. This provided URL also appears to be incorrect, as the app's manifest does not allow it to be run with 6.x.

I would suggest that the error be more specifically targeted at helping end users understand the problem and resolve it e.g. something like

This application depends on version 5.0.x of the shared .NET runtime, which is not installed on this machine.
The latest compatible version of the runtime can be found at https://aka.ms/dotnet-runtime?version=5.0&rollforward=minor&rid=osx.12-x64

Perhaps dotnet-tool could also detect the missing runtime and prompt users to install it upfront when installing the tool.

[ghost]

Tagging subscribers to this area: @vitek-karas, @agocke, @VSadov
See info in area-owners.md if you want to be subscribed.

Issue Details

---

A user of the .NET global tool vsmac-cli reported the following host error when trying to use the tool after installing it.

% vsmac
A fatal error occurred. The required library libhostfxr.dylib could not be found.
If this is a self-contained application, that library should exist in [/Users/[USER]/.dotnet/tools/.store/vsmac-cli/0.2.0/vsmac-cli/0.2.0/tools/net5.0/any/].
If this is a framework-dependent application, install the runtime in the global location [/usr/local/share/dotnet/x64] or use the DOTNET_ROOT environment variable to specify the runtime location or register the runtime location in [/etc/dotnet].

The .NET runtime can be found at:
- https://aka.ms/dotnet-core-applaunch?missing_runtime=true&arch=x64&rid=osx.12-x64&apphost_version=6.0.3

This error occurred because the app is framework dependent and the end user does not have the required framework installed. This is likely a common scenario for end users, yet the message is needlessly complex, with implementation details (libhostfxr) and irrelevant information (self-contained vs shared) that appears to be targeted at the developer of the app. This provided URL also appears to be incorrect, as the app's manifest does not allow it to be run with 6.x.

I would suggest that the error be more specifically targeted at helping end users understand the problem and resolve it e.g. something like

This application depends on version 5.0.x of the shared .NET runtime, which is not installed on this machine.
The latest compatible version of the runtime can be found at https://aka.ms/dotnet-runtime?version=5.0&rollforward=minor&rid=osx.12-x64

Author:	mhutch
Assignees:	-
Labels:	area-Host, untriaged
Milestone:	-

[richlander]

@elinor-fung

[filipnavara]

I can agree about the message but you also touched on a different issue - on macOS/ARM the global tools depend on the x64 version of the runtime. That's unintuitive and sometimes even quite harmful (eg. installing dotnet-sos installs extensions for x64 lldb). I suspect it was some last minute change because of signing issues because most .NET 6 previews didn't behave that way. @richlander do you happen to know if there's any issue related to that?

(Acquiring a tool through dotnet tool install would normally imply the runtime is already installed so this would not be an issue in the first place.)

[mhutch]

(Acquiring a tool through dotnet tool install would normally imply the runtime is already installed so this would not be an issue in the first place.)

It implies a runtime is installed, but not necessarily the one required by the tool, which is constrained by its roll-forward policy.

[richlander]

I suspect that this tool is cross the Intel/Apple arch boundary.

My guess is that dotnet tool being run via the Arm64 SDK and you have no x64 installation.

[filipnavara]

It implies a runtime is installed, but not necessarily the one required by the tool, which is constrained by its roll-forward policy.

I am well aware of the roll forward policy. It's customary to set it on dotnet tools.

I suspect that this tool is cross the Intel/Apple arch boundary.

That's my point. Look for example at the dotnet-sos NuGet. It has binaries for osx-arm64 and yet if you run dotnet tool install -g dotnet-sos on ARM64 macOS it will require the x64 runtime and run under it. That was NOT the case of the .NET 6 Previews, it was some late change. I'm just asking whether it's documented or whether there is issue filed for it.

(FWIW the dotnet tools don't distribute the host executable, neither does the vsmac-cli NuGet.)

[agocke]

global tools depend on the x64 version of the runtime.

This is news to me. I don’t think there’s anything at the host level which would constrain this — global tools should be creating the appropriate host for the platform.

The only exception is 3.1/5.0 apps, which don’t have an ARM64 version at all. If this isn’t a 5.0 app it sounds like a bug.

[filipnavara]

The only exception is 3.1/5.0 apps, which don’t have an ARM64 version at all. If this isn’t a 5.0 app it sounds like a bug.

vsmac-cli was likely compiled as net5.0 (by looking at the NuGet content). dotnet-sos which has package version 6.0.x seems to be still compiled against netcore3.1 (for backwards compatibility?). Then again, it has roll forward policy and osx-arm64 binaries, so I don't think the intent was to require x64 runtime.

Note: both packages are compiled as any; vsmac-cli doesn't seem to have the roll-forward policy, dotnet-sos has it.

[agocke]

Roll forward is about version resolution, which happens much later then dotnet tool installation, so that isn’t used to determine installation architecture.

I’ll leave it up to the tool-install team to decide if they want to make roll forward an installation concept as well.

However, for tools with native binaries (which may include SOS) there really isn’t another option — you have to have x64 installed.

[filipnavara]

Roll forward is about version resolution, which happens much later then dotnet tool installation, so that isn’t used to determine installation architecture.

I understand that. However, up until net6.0 GA the behavior was to install the platform specific host with no exceptions. Specifically, no exception for osx-arm64. I understand some reasoning about why the exception was put in place but I haven't seen any discussion about it. Hence why I ask whether it's documented or filed as an issue.

However, for tools with native binaries (which may include SOS) there really isn’t another option — you have to have x64 installed.

The problem is that the tool does contain osx-arm64 native binaries but they are never used with the x64 host. The whole tool is likely still targeting netcoreapp3.1 for backwards compatibility. So on arm64 host it never installs the correct LLDB binaries with the current behavior effectively making it quite useless (apart from a scenario where you use x64 lldb under Rosetta with x64 dotnet binaries running under Rosetta).

[agocke]

does contain osx-arm64 native binaries

This is a broken package. You cannot have OS X ARM64 native assets with a netcoreapp3.1 target in a supported way. FYI @tommcdon

[agocke]

However, up until net6.0 GA the behavior was to install the platform specific host with no exceptions

I didn’t know we installed a host at all pre 6.0. I think installing one now is more of an implementation detail, to cope with multiple architecture deployment. @marcpopMSFT to comment on that

[filipnavara]

This is a broken package.

How is one supposed to make it correct though? Specifically, how is one supposed to produce a global tool that installs on all supported .NET versions (or for simplicity, net5.0 and net6.0) and still use arm64 runtime if running on Apple Silicon Mac? Should the NuGet contain duplicate set of binaries with one of them in the net5.0 directory and the other in net6.0 directory?

[filipnavara]

@mhutch Sorry for tainting the ticket and going astray from the original issue (error message). I thought that there were already relevant issues about the global tool deployment and I'd just get redirected to them. I believe that in the case of vsmac-cli it would be beneficial if it run with the native runtime on osx-arm64 without going through the hoop of installing a package for non-native architecture.

[elinor-fung]

The behaviour where the global tools on macOS arm64 installing a x64 host is actually somewhat related to the error message being so bad. Because there is no runtime (or hostfxr) matching the architecture of the host, when that message is shown, we don't have much information about what exactly is being run, how it is configured, and what runtimes or versions it depends on. We should definitely reduce the complexity of the message and highlight the architecture that is required - something like:

You must install .NET to run this application.

Architecture: x64

Download .NET:
https://aka.ms/dotnet-core-applaunch?missing_runtime=true&arch=x64&rid=osx.12-x64&apphost_version=6.0.3

[agocke]

Should the NuGet contain duplicate set of binaries with one of them in the net5.0 directory and the other in net6.0 directory?

They would be multi-targeted, so they wouldn't be duplicated (they would reference 6.0 versions), but yes. Multi-targeting is the preferred way of supporting multiple frameworks.

[mhutch]

A bit tangential, but perhaps we should have a build warning for global tools that don't do major version roll-forward. I have it set on another of my global tools (dotnet-t4) but missed it on this one - it's pretty easy to do so.

[richlander]

I didn't see an answer to my question. It is true @mhutch that you have only Arm64 and no x64 installed? I am assuming but want to check.

[mhutch]

This isn't my machine, the error was reported to me (I don't have an M1 myself). I'll ask for details, but the following was also reported and may be relevant:

• downloading the project and trying to run it directly with dotnet run resulted in a similar error to the installed copy of the tool
• after changing the project to target net6.0, dotnet run worked
• after installing .NET 5, the installed copy of the tool ran successfully

This is consistent with the error in this particular case being related to a missing version of the runtime rather than a missing architecture, though I expect a missing architecture would result in similar errors.

[elinor-fung]

From the host side, this error should only be shown when we can't find a runtime (of the matching architecture) at all. That is why I'm thinking it is not just a missing version, but a missing runtime. When we find a runtime, but not a compatible framework/version, the error would have more information (like the framework and version that are missing).

after changing the project to target net6.0, dotnet run worked

This may be related to what @agocke was saying, where before 6.0, there was no ARM64 host, so the user would end up with the x64 host. But after switching to target net6.0, they get the ARM64 host.

[mcumming]

This isn't my machine, the error was reported to me (I don't have an M1 myself). I'll ask for details, but the following was also reported and may be relevant:

• downloading the project and trying to run it directly with dotnet run resulted in a similar error to the installed copy of the tool
• after changing the project to target net6.0, dotnet run worked
• after installing .NET 5, the installed copy of the tool ran successfully

This is consistent with the error in this particular case being related to a missing version of the runtime rather than a missing architecture, though I expect a missing architecture would result in similar errors.

cc: @richlander

It was my machine that I received this error on, here is a screenshot of my terminal session prior to installing the .Net 5 x64 runtime:

Interestingly enough the .Net 5 runtime is still not displayed when I run the dotnet --list-runtimes, I am guessing that is because my sdk's are all ARM64 and the .Net 5 runtime is x64.

[richlander]

That's giving us some of the info we need. We really need to know if you have an x64 .NET installed. Can you share a directory listing of /usr/local/share/dotnet?

One change we should consider is augmenting dotnet --info to show if there if a hostfxr exists for another global root (x64, arm64, x86). That would make troubleshooting conversations go faster.

We should also ensure all of our error messages include architecture, including when they mismatch.

[richlander]

Stepping back one step. The experience you are seeing is always going to be an error, by design. The goal here is to provide a clearer error that directs the user towards self-help. That's what the issue title describes already, so I'm not stating anything novel, just a reminder to anyone reading this.

[vitek-karas]

The dotnet run behavior is explained by this:

• If the app is targeting .NET 5 or lower, the SDK knows that there's no ARM64 compatible runtime for such version, so it inherently targets it to x64. So it produces x64 executable and runs that - which fails with the "bad" error above as there's no x64 runtime at all (if there was one, the error would be much clearer about missing version).
  • Overall this is by-design - the SDK never checks if the target TFM has a matching runtime installed. I agree the error message could be better for this case though.
• If the app is targeting .NET 6+ SDK will default to the architecture of the SDK itself - which in this case is ARM64 - that will produce ARM64 executable and run.

There are some difficulties with this approach:

• If the app allows Major roll forward, SDK "could" be clever enough to know that even though the app targets net5.0 it will run on 6 and this leave it as ARM64 - we currently don't do that.

I don't know how the dotnet tool picks the architecture for the apphost to generate, but I think it needs to have a similar logic in it with possibly similar problems.

The fact that .NET 5 runtime is not listed in dotnet --info is a known limitation currently - that command right now only lists runtimes of the matching architecture (so ARM64). You would need to run the x64 dotnet to get those runtimes (and sdks).

[richlander]

If the app allows Major roll forward, SDK "could" be clever enough

Ya. I'd like that, but I haven't heard anyone ask for it. I suspect that this space is too complicated for folks to connect all the dots that this could be a good/reasonable UX. Once we improve the roll-forward UX (dotnet/designs#259) then I think more people will use it. That said, the whole problem is going to disappear for most users when 3.1 and 5.0 go EOL (later this year).

The fact that .NET 5 runtime is not listed in dotnet --info

@elinor-fung and I talked about this. We're proposing a much simpler solution to that, just a boolean check of which architectures exist (and report the result in dotnet --info. It's also fine if the check reports false positives.

[filipnavara]

If the app allows Major roll forward, SDK "could" be clever enough

Ya. I'd like that, but I haven't heard anyone ask for it.

I would have asked for it...but it worked in the .NET 6 previews up until some moment. Admittedly it may have worked because it didn't check the TFM but it definitely changed at last minute.

[richlander]

It did "work" until late in the game, until the design was implemented. See: https://github.com/dotnet/designs/blob/main/accepted/2021/x64-emulation-on-arm64/x64-emulation.md#rely-on-implicit-rid-defaults

The old approach made the roll-forward experience much better, but it made the other experience of jumping arches effectively impossible.

[filipnavara]

It did "work" until late in the game, until the design was implemented.

I understand that. My point is that you likely didn't hear much complaints also due to the fact that the most active people were on the preview versions. I installed most of the global tools before the design was implemented and so they are already installed with the ARM64 app hosts. I only noticed the change after several months when I needed one more additional tool.

[richlander]

Fair enough.

We're definitely still open to improvements in this area. I assert that the design we have now has the right foundational behavior, so its a question of how we can approve the UX from there.

The easiest thing is to raise scenarios that don't work or are inconvenient.

[akoeplinger]

Similar issue where we hit problems with a dotnet tool missing the right shared framework, there was some pushback about making the rollforward option more discoverable but I still think this is the better "just works" option for a majority of the cases: dotnet/runtime#61019

[vitek-karas]

@akoeplinger I think the problem with global tools should not be solved in the host - host is for all apps, lot of the above problems are relatively specific to global tools. Also things like roll-forward are very focused on developers, the host needs to target "normal users" - so anything more complicated than "Please visit this URL" is probably too much (and yes, the current experience is unfortunately already too much).

I think the core of the problem is that dotnet tool install makes several assumptions which are problematic to say the least:

• The tool is applicable to the local machine (what if that tool is simply not meant to run on Linux... ???)
• The tool can run on any version of .NET (maybe with some minimum version)
  • Basically assumes that the tool has rollforward set to Major
  • Assumes that the tool will never break on new .NET
• The tool can run on the architecture/OS of the current machine - this goes to the problem with RIDs and so on, but in short, what if the tool simply doesn't want to support some combination (maybe because it can't)
• The tool better not want custom executable (like for example powershell), otherwise it's a mess

I personally don't know much about the dotnet tool install to be honest, but lot of the above assumptions seem to be part of the system in some way or form. And some of them are simply not viable for the tools we want to use this for.

I think the solution to this would start by having a clear declaration of the requirements for a given tool and the dotnet tool install validating such requirements. This can be .NET version, architecture and many more. Ideally the dotnet tool install would also be able to fix some of these - like installing new/older version of the runtime, installing different architecture, ...

Basically dotnet tool install tries to act a little like a package manager - without implementing most of the complexities which come with it.

[richlander]

Basically dotnet tool install tries to act a little like a package manager - without implementing most of the complexities which come with it.

+100. This is a CLI issue.

That all said. There is a simple workaround. Multi-target. That's what the EFCore folks did: dotnet/efcore#27807.

In the ideal experience, dotnet tool install would realize that the configuration you'd installed didn't make sense and would ask you if you wanted to:

• Install .NET 5 x64 (which BTW is EOL), or
• "Upgrade" the app to run as an Arm64 app with .NET 6.

Apps from Microsoft don't need any of that, however. They should multi-target, as efcore did.

Also, we improved the error messages including dotnet --info to enable better self-help.

https://devblogs.microsoft.com/dotnet/dotnet-apphost-improvements/

[richlander]

Also, should we start with better guidance on this scenario? I'm about to update our guidance on multi-arch systems. Should I include this scenario and suggest that the guidance I gave the efcore team (MT) is the best practice?

[dhduvall]

I ran into this (or some subset of it) today. I think some of this is new information; I hope it's useful.

I got into the situation originally with SDK 6.0.300 (6.0.5 runtime), but it's reproducible with the latest as of today—7.0.200 (7.0.3 runtime). I'm on an M1 Mac (arm64), and this was my experience:

$ dotnet --info
[ see bottom of comment ]

$ dotnet tool install --global coverlet.console
Skipping NuGet package signature verification.
You can invoke the tool using the following command: coverlet
Tool 'coverlet.console' (version '3.2.0') was successfully installed.

$ coverlet -h
You must install .NET to run this application.

App: /Users/duvall/.dotnet/tools/coverlet
Architecture: x64
App host version: 7.0.3
.NET location: Not found

Learn about runtime installation:
https://aka.ms/dotnet/app-launch-failed

Download the .NET runtime:
https://aka.ms/dotnet-core-applaunch?missing_runtime=true&arch=x64&rid=osx.13-x64&apphost_version=7.0.3

The instructions at the app-launch-failed URL just told me to install the .NET runtime, which I'd already done. I'd used the installer, so it should have been in the expected place, etc.

It took me a while to notice the x64 in that output, and realize that it wasn't complaining about the version (though when I first ran it, I didn't have 7.0 installed, so it was probably complaining about that, too), but the architecture.

What I didn't see mentioned in the discussion above was the --arch flag to dotnet install:

$ dotnet tool uninstall --global coverlet.console
Tool 'coverlet.console' (version '3.2.0') was successfully uninstalled.

$ dotnet tool install --global --arch arm64 coverlet.console
Skipping NuGet package signature verification.
You can invoke the tool using the following command: coverlet
Tool 'coverlet.console' (version '3.2.0') was successfully installed.

$ coverlet -h
Cross platform .NET Core code coverage tool 3.2.0.0

Usage: coverlet [options] <<ASSEMBLY|DIRECTORY>>
[ ... ]

The original discussion here is from some time ago, and I'm sure things have changed a bit since then, but it seems like much of the discussion has been about whether the tool is available at all for ARM64. But that certainly doesn't seem to be my problem. Instead, it looks like even when there is a choice, the installer makes the wrong one.

dotnet --info

.NET SDK:
 Version:   7.0.200
 Commit:    534117727b

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  13.1
 OS Platform: Darwin
 RID:         osx.13-arm64
 Base Path:   /usr/local/share/dotnet/sdk/7.0.200/

Host:
  Version:      7.0.3
  Architecture: arm64
  Commit:       0a2bda10e8

.NET SDKs installed:
  6.0.406 [/usr/local/share/dotnet/sdk]
  7.0.200 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.14 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.3 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.14 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.3 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

[baronfel]

Added this one to the list of things for the SDK team to tackle for the .NET 8 cycle.

[itisNathaniel]

Thanks @dhduvall, just been experiencing exactly the same issue on M1 with it trying x64 while trying to get dotnet-ef installed and working. Turned out it was the same and --arch got it working