Nuget CPM Lock file Proposal (#12409)

[CEbbinghaus]

After some Internal deliberation I have come up with a fairly straightforward design as to how a Central lock file could be implemented in Nuget. This addresses #12409 which is currently the 3rd highest rated issue.

Rendered Proposal

Appreciate any feedback <3

[CEbbinghaus]

@dotnet-policy-service agree company="WiseTech Global"

[DrNeil]

This would be super useful.

[CEbbinghaus]

What can we do to further progress this proposal?

[JonDouglas]

@CEbbinghaus Thank you kindly for starting a discussion on this proposal! Give me some time to look through it and get the team together to review it. I didn't see the notification earlier as I've been terribly busy with some other things. I apologize in advance for the ~2 month delay.

[CEbbinghaus]

I appreciate you taking a look at it now. Please do let us know if anything needs addressing

[kasperk81]

i love JSON as much as the next guy. but if Directory.Packages.props, NuGet.props and most of the files are XML based, wouldn't it be better to not mix stacks? historically lock files use .json format in various programming platforms, so that might be one defense for it (but not a very compelling one).

[yaakov-h]

The existing related files are already JSON - packages.lock.json and project.assets.json.

[CEbbinghaus]

Personally my preference is also xml but the nuget/dotnet team have already settled on a json based lock file and this change is supposed to be as small and simple as possible.

[purepani]

Is there a reason to make the lock file path configurable in the project? AFAIK, most package managers leave it unconfigured, and that way, the package manager wouldn't have to first read the configuration file first in order to install packages, only the lock file(which makes it easier to separate into steps).

[CEbbinghaus]

The package manager would need to read the Directory.packages.props file anyways as it is what enables CentralPackageManagement in the first place? (relevant docs)

We could get away with setting a property like <RestorePackagesWithLockFile>true</RestorePackagesWithLockFile> to enable CPM Lock files and implicitly assume the <NuGetLockFilePath> to be packages.lock.json but we can also achieve the same with a single property and a check $(NuGetLockFilePath) = ''. At the end it will be up to the maintainers to give feedback which way they prefer but the lock file path will need to be configurable in any case.

[purepani]

The package manager would need to read the Directory.packages.props
Yeah, but you can separate that into steps right? I'm imagining a use case of being able to separate it into layers:
You read the configuration file and can generate the lock file in one step, and then you can read the lock file and get the packages in a different step. The reason this could be useful is that you can write a completely separate program to read the lock file if you wanted to(one that assumes that you're using CPM), and you wouldn't have to read the configuration file to figure out where the lock file is.

I personally can't think of a great reason to not have the lock file at the top level of a repo(or at some fixed path relative to the top level), and I can think of more reasons to avoid being able to configure it(Easier to make external tools to consume the lock files and process them separately, avoiding conflicts between, say, a git submodule with it's own lock file that's configured to generate this lock file, etc.).

Given that, as far as I'm aware, most package managers(mostly python and node ones) that I've used have a fixed path for their global lock files, I thought I'd at least bring it up.

[yaakov-h]

You would need a way to define the "top level" location. This allows for that.

Keep in mind that not every repo-project relationship is 1:1, particularly in a monorepo situation where you might want a small handful of "top levels" rather than only one.

[purepani]

It should just be in the same folder as the Directory.Packages.props file, the same way that the "top level" of a nodejs project is where the package.json is.

[CEbbinghaus]

Sorry i feel you might have misunderstood. This is exactly the case @purepani the <NuGetLockFilePath> specifies the lock file for the CPM rather than each individual project. This will be located relative to the Directory.Packages.props at the root of the project.

Now that you mention it we will likely need a property like RestorePackagesWithLockFile but specifically to enable CPM lock files but the lockfile will be located at the root NOT under individual projects

[purepani]

I see so this property is meant for each individual project rather than just the top level of the projects; that makes way more sense.
The way pnpm solves this is by symlinking the dependencies into each workspace after doing a pnpm install at the top level. If you only wanted some dependencies, you could just install it for that group. In this case, it is just the dependencies and not the lock file, but you could certainly symlink the lock file if you wanted(though the only advantage of that approach would be that you wouldn't have to move to the top level every time you wanted to install a package).

(note: I only mentioned pnpm specifically because I know how it works a bit better. Presumably other node package managers work similarly)

[JonDouglas]

I appreciate you taking a look at it now. Please do let us know if anything needs addressing

Just an update, we have various folks out and were busy with our recent conference. I will try to get this scheduled once people are back from summer vacations.

[CEbbinghaus]

can we add a nostale tag?

[nkolev92]

Thanks for starting the write-up.
I've left some comments about some technical/functional challenges we'd need to figure out before taking this further.

[nkolev92]

I think something worth mentioning here is the concept of transitive pinning: https://learn.microsoft.com/en-us/nuget/consume-packages/central-package-management#transitive-pinning, which allows you to control the transitive dependencies by effectively elevating them to top level.

[yaakov-h]

Transitive pinning only helps when you recognise that you've gotten into this situation and want to do something about it. It doesn't prevent you from getting into a conflict in the first place unless you're very eager and go and manually specify every individual transitive dependency and keep track of them when updating packages.

[nkolev92]

That's correct. It's basically exhausitve transitive pinning where all packages are pinned.
The lock file you're asking for would effectively be that, but potentially in a more automated fashion.

[nkolev92]

There's a technical complexity here since a property value doesn't get path resolved, so knowing where it comes would be required.
The simpler approach might be the already mentioned just making the file live next to the directory.packages.props.

[CEbbinghaus]

For flexibility I prefer the idea of having the lock file path configurable which also tracks consistently with lock files on a project. The better solution is probably setting the default property to be next to the Directory.packages.props and instead offering (if it doesn't already exist) a $(ThisCurrentFilePath) MSBuild property which can be used to set the file path relative to the Directory.packages.props rather than something relative to the project.

[nkolev92]

Yeah, it's all doable, we'd just need to be specific to avoid confusion.

[nkolev92]

The biggest challenges delivering this feature are the preexisting msbuild paradigms and we're not really tackling that here.

We need to dive deeper into the technical complexities to understand the feasibility of simply having a lock file.

How would the scope of the msbuild based restore operation be defined? Currently restore is either project based or solution based. Projects don't know about all other projects.

Here's a practical example:

You have 2 projects and 3 packages.

Project 1:

<PackageReference Include="A" />

Project 2:

<PackageReference Include="B" />

DPP

<PackageVersion Include="A" Version="1.0.0" />
<PackageVersion Include="B" Version="1.0.0" />

A 1.0.0 -> C 1.0.0
B 1.0.0 -> C 2.0.0

Without pinning:

• Project 1: A 1.0.0 and C 1.0.0
• Project 2: B 1.0.0 and C 2.0.0

These projects aren't necessarily correlated so the only way for them to know that they're conflicting with C is to either:

• Consider what each project references each restore
• Enforce version declaration for every package version encountered.

Today, a project only knows about itself and it's closure, so this would be a departure in how restore/build work, one that'd come with some perf considerations.

The 2nd idea could theoretically be solved with transitive pinning.

You can declare C 2.0.0 in DPP, enable transitive pinning and then the versions would be resolved would be equivalent.
You'd basically need to exhaustively declare all versions.

Within the current toolset, The central lock file as proposed is really a more algorithmically advanced transitive pinning that gets applied across all projects.

Another challenge is package creation for any projects that have a central lock file.

NuGet works really hard to make projects and the package versions of them behave as closely as possible.
What this means is that every PackageReference you have becomes a dependency in a generated package.
To avoid runtime issues, transitive pinning elevates those transitively pinned packages to top level dependencies. The practical example is Project 1 from above, where it could unwittingly take a dependency on C 2.0.0, and then if it gets published as a package with only its direct dependencies lead to runtime failures later.
I understand that's not really a relevant scenario here, but it is a concept that the design needs to account for.

[yaakov-h]

the central lock file as proposed is really a more algorithmically advanced transitive pinning that gets applied across all projects

Isn't that exactly what a lock file does, by specifying the entire dependency graph and the versions used of each dependency?

The practical example is Project 1 from above, where it could unwittingly take a dependency on C 2.0.0, and then if it gets published as a package with only its direct dependencies lead to runtime failures later.

I assume we would need to keep track of which dependencies are direct and which are transitive, and dotnet pack would only look at direct dependencies when building a package. A transitive dependency can always be elevated to a direct dependency.

[nkolev92]

Isn't that exactly what a lock file does, by specifying the entire dependency graph and the versions used of each dependency?

There's a key distinction. All version resolution is project based.
Lock files don't change the resolution. They skip it.

The idea for central lock files makes the version resolution more involved and requires to account for transitive packages declared from all projects, not just the current one.

You're not just asking for a file, you're asking for a completely new resolver. Beyond scoping, the resolver part is the most challenging technical part.

I assume we would need to keep track of which dependencies are direct and which are transitive, and dotnet pack would only look at direct dependencies when building a package. A transitive dependency can always be elevated to a direct dependency.

"dotnet pack would only look at direct dependencies when building a package" -> This doesn't work in all scenarios.
Here's an example:

Project 1 -> Package A 1.0.0 -> Package B 1.0.0
Regular resolution leads to A 1.0.0, B 1.0.0.
When the project is packaged, it only needs A 1.0.0 as a dependency, since at minimum B 1.0.0 will be resolved due to A's dependency.

Central lock file example could bump package B to 2.0.0.
Project 1 code ends up coding against B 2.0.0.
If the Project 1 package ends up declaring A 1.0.0, then when anyone installs that package generated from project 1, they'd get B 1.0.0, because no one knows about 2.0.0 in the Project 1 graph, it's there due to how other projects work. If we don't elevate B 2.0.0 to a direct dependency, then the consumer of Project 1's generated package would end up with runtime failures.

[nkolev92]

Some general food for thought for the concept of centralized lock files.

In general, you probably only need to consolidate the versions being used across your runnable projects. Your services, your tools etc, not across your libraries as well (and never downgrade ofc).
Just because you make your projects reference a specific version, that doesn't mean they'll all work. You could be referencing dependencies that reference different versions of a particular package, so consolidation is naturally going to happen in that case.

There's of course the theory for handling breaking changes in dependencies, but that'd only work if you control all libraries that reference a package with a breaking change, in which case regular CPM would work just fine.

[CEbbinghaus]

I prefer the idea of promoting any transitive dependency at a version outside the already resolved range into a primary dependency so that any downstream library consumers don't need to think about it

[nkolev92]

We need to add the mechanics of version resolution in the unresolved questions.

[JonDouglas]

Hey all, adding a quick high-level comment on this proposal as we did a quick team review yesterday.

First, I just want to say, thank you so much for this proposal! The goal of introducing a central lock file concept seems well received based on the number of reactions in the short period of time since it was introduced here and between the linked issue.

Here are some of the challenges we're looking through:

• Feasibility of implementing central lock files, challenges of regenerating lock files, and a lack of a mechanism that represents the entire project tree.
• Merge conflicts, challenges of updating package versions w/ the central lock file concept.
• Significant changes in the tooling and workflows to accommodate central lock files.

Upsides:

• Ensuring consistency in dependency versions across multiple solutions / repository.
• Recognizing the community's evolving needs for better dependency management in monorepos and multi-solution projects.
• Evolving beyond a project and solution level

We encourage further discussion and exploring this topic further while we continue to assess feasibility with others teams (i.e. MSBuild, etc) and how this would be implied throughout the .NET ecosystem.

Please continue to comment, raise your support/disapproval, and provide any prior art to help us investigate further!

[yaakov-h]

<generic activity here>

[ajeckmans]

Bad bot! :)

[CEbbinghaus]

Can we just give this Item a No-Stale flag so we don't have activity for the sake of activity. People will keep sending a message whenever the bot tries to stale it