Add proposal for RID-specific .NET Tool packages

[dsplaisted]

No description provided.

[jkotas]

The exact same problem exists for regular nuget packages too. Any chance we can generalize the solution to solve the problem for all types of nuget packages, and not just tools?

Related question: Is the downloader for the proposed dependent packages going to be implemented in nuget or in the SDK?

[akoeplinger]

NuGet/Home#10571 is an issue about the problem for regular nuget packages.

I'd be very happy if this could replace the runtime.json hack.

[dsplaisted]

Related question: Is the downloader for the proposed dependent packages going to be implemented in nuget or in the SDK?

This is implemented in the SDK, but it calls NuGet APIs to download the packages. That is how tools already work, this proposal updates the logic so that once the SDK downloads the primary package it will see that there's a RID-specific package to download and download that one too.

Any chance we can generalize the solution

Lots of people have had this feedback and we also discussed it in a meeting yesterday. I don't think we would want to use the same system for tools as for RID-specific package dependencies. I will add a section to the design explaining this, and I'm sure we'll also discuss it more.

[ericstj]

Also NuGet/Home#1660

[rolfbjarne]

However, there might be other files in the same folder that the tool depends on, so we don't currently plan to do this.

Here's an example of something that wouldn't work:

1. A tool using ClangSharp would be an excellent test case for a rid-specific tool package, because the native clang dependency is big.
2. A tool using ClangSharp would have a dependency on a native dynamic library (dll/dylib), and as such copying just the executable would not work.

[joelverhagen]

Why have optional Version? Won't it be known at pack time? I am unsure why we should allow a missing version at all.

[dsplaisted]

If you are building a NativeAOT app, then you mostly have to build for each Runtime Identifier on a separate machine. So depending on how your build/pipeline system works, you might not have the same version for all of the different packages.

[joelverhagen]

If the version is not same for all of the different packages, how would the top level tool package come to know about the list of (RID, version) tuples, i.e. to produce the DotnetToolSettings.xml file?

I see the example .csproj XML:

<ItemGroup>
	<ToolPackageRuntimeIdentifier Include="win-x64"/>
	<ToolPackageRuntimeIdentifier Include="linux-x64"/>
</ItemGroup>

This seems to be the non-NativeAOT case, where the whole graph (3 packages in this case?) is produced on one machine.

If the RID-specific packages are being produced on another machine, there seems to be a fan-out then collect type flow.

I'm trying to visualize how the differing version numbers will flow from one build step to the next. It feels like a bit of a mess if the version numbers are allowed to diverge. If we say "all version numbers must match" then there is no need for build steps to happen in a certain order or flow information (RID + selected version number for that RI) from one build step to the next. I see how it can be technically possible but I wonder what the .csproj will look like to handle the split-up, RID-specific build steps.

[dsplaisted]

The primary package would need to be built after all of the rid-specific tool packages, so the version numbers from those builds would be inputs to the primary package build. I don't know how you'd flow the version numbers from one pipeline to another, but at the project level you could pass those versions as properties and it would look like this:

<ItemGroup>
	<ToolPackageRuntimeIdentifier Include="win-x64" Version="$(ToolPackageVersionWin64)"/>
	<ToolPackageRuntimeIdentifier Include="linux-x64" Version="$(ToolPackageVersionLinux64)"/>
</ItemGroup>

I think it's a good idea for us to support this if people need it. If they don't then all the versions can match, and everything is simpler. But if they don't have a good way to have the build numbers on different architectures all match, then it is good to have an escape hatch for them.

[joelverhagen]

Package type has versioning. Currently DotnetTool is the default 0.0 version. If it is bumped to DotnetTool, 2.0.0 then they should still be searchable by DotnetTool but also have a persistent marker that could be later used for filtering, without having to crack open the package again to parse the DotnetToolSettings.xml. Right now we only have package type name searching enabled but we could enable version filtering in the future.

Conceptually, the package type is moving to version 2 as shown in your XML.

[dsplaisted]

Interesting. I'm not sure we knew that you could version package types when we originally implemented tools, or maybe at that time it wasn't possible to version them. Since old SDKs don't look at the package type we'll have to update the version in the tool manifest anyway so that old SDKs will error out when trying to install a newer tool. So I'm not sure there's much extra value in also updating the package type version.

[joelverhagen]

It would allow NuGet.org to filter these by version, and analyze their adoption in NuGet Insights (separate from V1 tools). It could allow customized UI on NuGet.org to show the RID-specific dependencies (after analyzing DotnetToolSettings.xml). In other words, it's the "NuGet" way of versioning package types.

Package type versioning has been there since the beginning. I made it :)

It's there to use if you want it. I think it would be good to have these new packages marked differently, in a nuget-specific way if possible.

[joelverhagen]

Will the RID-specific package be validated to have DotnetToolRidPackage? Or will any existing package be accepted and treated as a tool package?

The relationship is only 2 layers deep, right? 1 DotnetTool to N DotnetToolRidPackage, and no DotnetTool -> DotnetTool -> DotnetToolRidPackage?

[dsplaisted]

The RID-specific package won't have DotnetToolRidPackage in its manifest, that's only for the primary package to redirect to the RID-specific packages. Theoretically a tool package could redirect to any other tool package. There's currently only layers though, it wouldn't keep following redirects.

We might also decide we want to support tool redirects. I think that might be useful as part of the one-shot / dnx / dotnet tool exec work in order to hide the difference between a tool package name and its command name. But that's not part of a proposal yet.

[joelverhagen]

Okay, I think I was confused at first by the use of the term "manifest". In NuGet world we sometimes refer to the .nuspec as the package manifest (this is where package types are expressed).

What I am understanding is that the .NET SDK will only follow one layer of edges in the graph (tree, at this point).

EXISTING BEHAVIOR: install tool package A -> this downloads and operates on A, a single .nupkg, with no package type validation.

NEW BEHAVIOR (in addition to existing behavior): install tool package X -> this downloads and operates on two .nupkgs: X and Y. X has a pointer to Y in the DotnetToolSettings.xml. There will be no package type validation for X or Y.

If Y happens to have its own RID-specific pointer to Z it will not be followed (no recursive graph walk).

[joelverhagen]

Is there a GitHub issue with more context on the "why" such as customer comments? Just curious!

[dsplaisted]

I don't know if there's an issue for this. This makes tools better overall, and more specifically allows tools to be compiled with NativeAOT, and more specifically could help make it easier to distribute an AI MCP implemented in .NET and compiled as NativeAOT.

[joelverhagen]

Will package source mapping, vulnerability auditing, signature validation and policies (etc) all apply to the RID-specific package? I think what I'm getting at, is: will security features that are added to restore happen here too automatically?

[dsplaisted]

The download of the RID-specific package will be handled the same way tool package downloads are already handled today. We call NuGet APIs to download the package. Depending on the feature (security or not), we might not support it by default without a change to the SDK code. When package source mapping was originally introduced we didn't support it for tools (or .NET SDK workload downloads, which also use the same APIs), and we had to update the SDK to support it. Signature validation may already be handled by the APIs we call. I think vulnerability auditing doesn't currently apply to tools, as they aren't referenced as packages in projects and don't go in the assets file.

So in general, whenever NuGet does a security feature we should think about how or if it applies to .NET Tools, which are a different way of consuming packages. However, adding support for RID-specific packages doesn't change anything here, as the RID-specific packages will be handled the same way as other tool packages are. (Note that we also have PackageDownloads in the .NET SDK, which are a bit more similar to normal NuGet package consumption but are still a different case to be considered.)

[joelverhagen]

Okay, thanks for clarifying. It's unfortunate that some security features don't work for current .NET tools. That seems like a problem but I see how it's orthogonal to this change. I think the reason I asked is because this feature feels like a mini restore operation with a relatively shallow graph. Since there are now transitive nodes it may be harder for users to reason about the package that they use and depend on.

Maybe the only bit of feedback here is to consider in the design how users can discover and audit the RID-specific package with tooling, similar to how we have dependency package tooling.

[dsplaisted]

I thought I responded to this but it looks like it got lost or I never submitted the comment.

The RID-specific packages will be handled the same way as tool packages currently are. Whether security (or other) NuGet features apply to tool packages depends on how the feature is implemented. We don't run restore, we call NuGet APIs to download a package. In some cases that may get us new features by default, in other cases it may not. For example, when package source mapping was introduced, we didn't immediately support it for tools (or for workload packages, which are also downloaded without restore). We had to update the downloader code in the .NET SDK to support the new feature.

I think NuGet auditing doesn't apply at all to tools because tools are not expressed as PackageReferences and don't show up in your assets file.

So when new security features are added, we should consider whether they should apply to .NET Tools, and if so whether we need to update the .NET SDK to enable that. But again, RID-specific tools don't really change this, because the RID-specific packages are downloaded the same way as existing tool packages.

[NinoFloris]

It seems like some concept of associated packages should instead be added to nuget packages proper.

If the package manifest could hold a list of associated package entries (e.g. {packagetype, name, optional? version range}) nuget could provide common authoring tooling and insights, and it would not be entirely opaque to nuget.org either.

Entries with certain package types could have some builtin resolution behavior or otherwise be ignored during restore. A spec like this would then define the custom resolution rules for the their package type.

As libraries have the exact same issue with rid splitting it strongly suggests things should not be tied to a custom file and schema that nuget does not understand. Sharing this part of the investment would make it much more likely we'll actually be able to get the library rid splitting issue solved as well.