Skip to content

[Spec] PackageDownload support

Jump to bottom
Nikolche Kolev edited this page Apr 5, 2019 · 1 revision

Package Download support

Issue

7339 - Support for download only packages scenario

Problem Background

Historically .NET Framework packs used to be separate installers. If you wanted to target .NET 4.6.2, you needed to install the .NET 4.6.2 targeting pack prior to building. In .NET Core 1.x & 2.x, the framework was all converted into packages. That evolved over the years leading to multiple meta packages with large graphs (Microsoft.NETCore.App, NETStandard.Library etc.). These meta packages were "auto-referenced" in the context of the NuGet graph. Each approach had it's pros and cons illustrated in the following design https://github.com/dotnet/designs-microsoft/pull/38.

A new hybrid approach, described in the spec linked above, for acquiring framework assets will be introduced in .NET Core 3.0, that leverages the strengths and eliminates some weaknesses of the previous 2 approaches.

Who are the customers

All .NET Core customers.

Requirements

  • Download only packages will not affect the project graph in any way, and as such are not written in the assets file. 1*
  • It should be possible to specify multiple versions of the same package ID.
  • Dependencies need not be downloaded. (For the purposes of the original use case, all framework packages will be authored without any dependencies)
  • Only the exactly specified version is downloaded.
  • Should have a no-op check similar to what we do today with installations in the global packages folder.

Solution

We will introduce PackageDownload, a new package defining item type that's slightly different from PackageReference. PackageDownload packages will not affect the dependency resolution graph. The PackageDownload items will be considered bound to the project and as such will affect the no-op restore. The errors and warnings generated by the PackageDownload items, analogous to all project restore warnings, will be written in assets file. A failure to download a PackageDownload package, is a full restore failure.

Implementation

** This feature will work with CPS/SDK based PackageReference only (CLI & Visual Studio).**

Package declaration

There are few fundamental differences between PackageDownload and PackageReference.

  • PackageDownload does not impact the project package graph.
  • PackageDownload is not transitive, the PrivateAssets metadata is irrelevant.
  • PackageDownload does not involve any assets selection, so the ExcludeAssets/IncludeAssets is irrelevant.
  • PackageReference supports version ranges while PackageDownload only works with exact versions.
  • The IsImplicitlyDefined is not relevant in the PackageDownload case.

Because of the above, we want to introduce a new item rather than re-use the existing PackageReference item.

To reiterate, we define a new item as following:

  • Duplicate Ids are allowed
Attribute Mandatory Description
Version Yes A version range equivalent to the PackageReference version. Initially only exact (ex. [1.0.0]) versions are allowed.

Error handling

If NuGet is not able to find the available packages NuGet will add an error message. There is not easy way for NuGet to communicate the operation status back to the SDK. The SDK would likely need to add their own error message. The messaging needs to be coordinated for the best user-experience. It's not worse than restore today. NuGet will consider adding new error codes for PackageDownload failures to allow for better experience from the SDK side.

Assets file changes

Initially, the only change to the assets file will be adding the PackageDownload items in the Project section of the assets file. The exact change. Open Question 3

PackageDownload and trust policies

The security requirements for these packages are the same as the user-packages restore. Meaning, if someone has client policies configured, they will have to add Microsoft to their trusted authors/owners for this restore to work.

PackageDownload repeatable build

In 15.9, the repeatable build feature was introduced, with project-based lock files. With user-packages, now there is a guarantee that the package you use on your local box is the same as the package you use on the build machine. Since the PackageDownload items are transient (in the targeting/runtime packs use-case they differ from machine to machine at the least), as such they will not be added to the lock file.

We can consider adding a new metadata to the download only package reference to communicate the same intent for these packages. The discussed approach is to add a new metadata on the PackageDownload element and for the SDK to provide this information explicitly. Tracking issue

Clarifications

Clarification 1

Through the numerous discussions this requirement was amended. By writing the package in the assets file, it will be easier for the IDE to display the node in the dependency tree.

Clarification 2

  • Notifying VS that new download-only packages were downloaded
    • Projects which don’t find targeting packs in “Program Files” should always emit “PackageDownload” items (and not also check in the global packages folder), so that if multiple projects in parallel need to download the same packages, VS can get notified for all of the projects that need new builds because they depend on updated package downloads.

Open work items

  • The Targeting/Runtime packs will both need a package type. That package type might be the same or different. Tracking issue

    • For future package type work it would be useful to have a checklist that defines all the characteristics of every known package type.
    • Guidance on how PackageDownload should be used. The lack of first class repeatability could be a problem.

  • PackageDownload and repeatable build. Tracking issue

Open Questions

Question 1

During the internal NuGet discussions, a point was raised to use _PackageDownload instead of PackageDownload. This to specify that it's not ready for public use as it's targeting a very specific scenario.

Question 2

How does PackageDownload tie in with a future download command. Is that an MSBuild target? Dotnet.exe?

Question 3

The dependencies (PackageReferences) are currently listed as:

"dependencies": {
    "a": {
        "version": "[1.0.0, )",
        "autoReferenced": true
    }
}

Option 1:

{
    "downloadDependencies": [
        {
            "name" : "a", 
            "version" : "[1.0.0, )"
        },
        {
            "name" : "a",
                "version" : "[2.0.0, )"
        },
    ]
}

Pros:

  1. Simple, one object, one reference.

Cons:

  1. Allows full duplicates

Option 2:

{  
   "downloadDependencies":[  
      {  
         "y":{  
            "version":"[1.0.0, )"
         }
      },
      {  
         "y":{  
            "version":"[2.0.0, 2.0.0]"
         }
      }
   ]

Pros:

  1. Similar to how we represent dependencies, but 1 level deeper.

Cons:

  1. Does not read as easy as option 1 or dependencies.
  2. Allows full duplicates.

Option 3:

{  
   "downloadDependencies":{  
      "y":{  
         "[1.0.0, )":{},
         "[2.0.0, )":{}
      }
   }
}

Pros:

  1. Does not allow full duplicates.

Cons:

  1. It is weird to have version ranges as a key.
  2. It will add another level of depth when we add new attributes.

Future work

  • We need to understand the implications of PackageDownload and the tooling such as:

    • Dotnet list package
    • Dotnet add package

Non-Goals

  • The publish scenario. When doing a self-contained publish, when runtime packs are specified as PackageDownload, the assets file will be stomped. This will mean that there will be 1 extra restore build after each publish. q
Risky cases
  • Installing SDK which uses a patched targeting pack after a successful build (where the next restore will be no-op)
  • Installing SDK while build is running (or VS is open)
  • Switching branches
  • Targeting pack for optional framework could fail to be downloaded, when that targeting pack isn’t actually used by your project

Alternative approaches considered

  • PackageDownload procedure running along side restore with different characteristics.
  • Runs alongside each user-packages restore.
  • It can no-op independently from the user-packages restore.
  • A project restore was considered complete only when both the user-packages and the download only packages have been installed.

The concerns of this approach were the following:

What happens if the user-packages restore no-ops, but there is a download only package that gets downloaded?

  • There has to be a way for NuGet to communicate to the project system what it download/whether it was downloaded. This is important for the dependency tree experience.
    • There should be a node in the dependency tree for each framework reference
    • If targeting pack isn’t available, there should be a warning icon on the corresponding node in the dependency tree
    • Associate warning icon in dependency node with warning message that is actionable (or at least understandable)

The 2 options discussed were adding the targeting pack download information in the project.assets.json and a completely new PackageDownload.assets.json file that the project system can watch. project.assets.json The pros of the project.assets.json approach are that it's only 1 file to watch, but it would lead to a weird matrix based on how/when the restore was triggered. With a new file, it's a new file to write/read and manage.

On other one hand, this approach does not have the caveat of the current solution, since the runtime packs do not affect the assets file. Overall this added significant complexity to the whole implementation.

  • AfterTargets hook for restore. This works just fine from the command-line, but introduces an ordering problem in VisualStudio.

    • Currently the trigger for a design time build after restore is the assets file write. In this approach, we'd either need to introduce another "trigger", which would be expensive, or have to coordinate the assets file write with the download of the targeting packs.

  • BeforeTargets restore. Similar issues to the above approach.

  • An asynchronous restore like process that's triggered only when required. Similar to the previous approaches, there is a design time build trigger issue. Furthermore there is no mechanism to hold of nomination after the evaluation has completed.

References

Contributing

What's Being Worked On?

Check out the proposals in the accepted & proposed folders on the repository, and active PRs for proposals being discussed today.

Common Problems

Clone this wiki locally