From 725130614f8f5211267437af0eb66cccab180830 Mon Sep 17 00:00:00 2001 From: CEbbinghaus Date: Mon, 4 Mar 2024 12:04:10 +1100 Subject: [PATCH 1/4] Added first iteration of RFC for Central Lock Files --- accepted/2024/repo-level-lock-file.md | 83 +++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100644 accepted/2024/repo-level-lock-file.md diff --git a/accepted/2024/repo-level-lock-file.md b/accepted/2024/repo-level-lock-file.md new file mode 100644 index 000000000..632f2bd06 --- /dev/null +++ b/accepted/2024/repo-level-lock-file.md @@ -0,0 +1,83 @@ +# ***Repository Level Lock File*** + +- Author: [@CEbbinghaus](https://github.com/CEbbinghaus) +- Issue: [Nuget/Home#12409](https://github.com/NuGet/Home/issues/12409) + +## Summary + +Applications consisting of multiple solutions (using assembly references between projects) Cannot control their transitive dependencies by using Nuget. The current lock file is on a per-project basis rather than an application/repository basis. Given a `Directory.packages.props` file, each project could be resolved with different versions of transitive dependencies, which a single central lock file would solve, by allowing consistency between package versions across multiple solutions. + + +## Motivation + +Monorepositories with multiple solutions utilizing assembly references that deploy all their code together as a single application are unable to use Nuget (with or without CPM) due to transitive version conflicts. This is due to nuget being able to resolve different versions of transitive dependencies for each individual build, as they occur in isolation from one another. A single central lock file allows all transitive dependencies to have locked versions defined at the root, thereby eliminating third-party dependency management software such as Paket. + + +## Explanation + +### Functional explanation + + + + +The `Directory.packages.props` now uses the`[path]` property within a `` to specify its lock file path. It becomes the source of truth for all builds. Project level lock files are no longer required unless a package version is overridden. + +```xml + + + true + lockfile.json + + + + + +``` + +This lock file is generated during any restore operation during the evaluation of the `Directory.packages.props` file. This allows the central lock file to be generated by any project as all of the dependencies are isolated. + +## Drawbacks + + +It complicates Nuget restoring as it will require a separate dependency resolution step exclusive to the `Directory.packages.props` file. + +## Rationale and alternatives + + + + + +Some alternatives exist. Namely: +* **Solution-based lock files** + This has the upside of giving us a comprehensive list of all projects that make up the lockfile. As such, the lockfile can be optimally generated, taking every dependency of every project into account, and doesn't require project-specific dependencies. The downside is lock files are defined in any of the multitude of solutions within a monorepository. This leaves the problem of syncing transitive dependencies between multiple solutions within the same repository. It also requires all restores to happen at the solution level, preventing building singular projects simultaneously. + +* **Using Microsoft.Build.Traversal** + `Microsoft.Build.Traversal` could create a `.csproj` that references all projects under it. As such, it would have the context of every package as defined by the `ProjectReference`. This would then act as the restore project that, in turn, sorts out all dependencies and generates the most optimal lock file. The downside is that only this project can be used to restore packages since the lock file would only apply to it, making it impossible to restore single projects. + + +## Prior Art + + + + + + +The primary example of prior art for a centralized lock file is [Paket](https://fsprojects.github.io/Paket/) built around this exact concept. It uses a `paket.dependencies` file, similar to the `Directory.packages.props` file from Nuget. It allows for 'groups' to pull two identical dependencies in different versions as each group is resolved independently. It then uses a `paket.lock` to lock the dependencies at the root (right next to the paket.dependencies), which pins all primary and transitive package versions for all projects. A project then includes a package by listing its name in the `paket.references` file in the csproj directory. + +We can draw many parallels between Paket & Nuget (with a central lock file) function. For one, we define all the primary packages & versions at the root. Projects define which specific packages they depend on within the relative projects. Where paket would use groups to include dependencies at different versions, NuGet uses the `VersionOverride` property within the csproj. This is the primary difference in how locks are handled for version differences since Paket utilizes groups and writes the locked package versions into the `paket.lock` file. Nuget (with central locking) writes all the pinned versions into the local csproj lock file. + +NodeJS has the concept of [Workspaces](https://docs.npmjs.com/cli/v7/using-npm/workspaces#skip-to-content) which act more like Solutions since NodeJs only has the concept of Projects. However a Workspace does allow for all packages to be resolved to the same version, ensuring no version conflicts between different projects within the same workspace. + +## Unresolved Questions + + + + + +* How would the `Directory.packages.props` dependencies be separated out from the project dependencies to generate the lock file? +* How does the publish command handle this as it doesn't have full context of all packages. (Could be solved by the restore always running with --locked-mode) +* How will a project deleted from the repository get deleted from the lock file. + +## Future Possibilities + +Using the `TargetFrameworks` property within the `Directory.packages.props` file could narrow down the frameworks resolved for the provided packages. This can simplify the lock file and reduce some unnecessary dependency resolution. From 28ffc93e3bc62e6523f07c783b4c9023012f7bad Mon Sep 17 00:00:00 2001 From: CEbbinghaus Date: Wed, 6 Mar 2024 16:20:24 +1100 Subject: [PATCH 2/4] Added extra unresolved question regarding nested files --- accepted/2024/repo-level-lock-file.md | 1 + 1 file changed, 1 insertion(+) diff --git a/accepted/2024/repo-level-lock-file.md b/accepted/2024/repo-level-lock-file.md index 632f2bd06..8741fcebd 100644 --- a/accepted/2024/repo-level-lock-file.md +++ b/accepted/2024/repo-level-lock-file.md @@ -77,6 +77,7 @@ NodeJS has the concept of [Workspaces](https://docs.npmjs.com/cli/v7/using-npm/w * How would the `Directory.packages.props` dependencies be separated out from the project dependencies to generate the lock file? * How does the publish command handle this as it doesn't have full context of all packages. (Could be solved by the restore always running with --locked-mode) * How will a project deleted from the repository get deleted from the lock file. +* How will Lock files be generated with nested `Directoy.packages.props` files. ## Future Possibilities From 9d5bc8fc45c93c3f6ffd7a92adcf082d7a5be7c1 Mon Sep 17 00:00:00 2001 From: CEbbinghaus Date: Wed, 20 Mar 2024 13:48:06 +1100 Subject: [PATCH 3/4] Applied Suggestions Co-authored-by: Yaakov --- accepted/2024/repo-level-lock-file.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/accepted/2024/repo-level-lock-file.md b/accepted/2024/repo-level-lock-file.md index 8741fcebd..fc306c68a 100644 --- a/accepted/2024/repo-level-lock-file.md +++ b/accepted/2024/repo-level-lock-file.md @@ -5,12 +5,12 @@ ## Summary -Applications consisting of multiple solutions (using assembly references between projects) Cannot control their transitive dependencies by using Nuget. The current lock file is on a per-project basis rather than an application/repository basis. Given a `Directory.packages.props` file, each project could be resolved with different versions of transitive dependencies, which a single central lock file would solve, by allowing consistency between package versions across multiple solutions. +Applications consisting of multiple solutions (using assembly references between projects) cannot control their transitive dependencies by using Nuget. The current lock file is on a per-project basis rather than an application/repository basis. Given a `Directory.packages.props` file, each project could resolve different versions of transitive dependencies, which a single central lock file would solve by allowing consistency between package versions across multiple solutions. ## Motivation -Monorepositories with multiple solutions utilizing assembly references that deploy all their code together as a single application are unable to use Nuget (with or without CPM) due to transitive version conflicts. This is due to nuget being able to resolve different versions of transitive dependencies for each individual build, as they occur in isolation from one another. A single central lock file allows all transitive dependencies to have locked versions defined at the root, thereby eliminating third-party dependency management software such as Paket. +Monorepositories with multiple solutions utilizing assembly references that deploy all their code together as a single application are unable to use Nuget (with or without CPM) due to transitive version conflicts. This is due to NuGet being able to resolve different versions of transitive dependencies for each individual build, as they occur in isolation from one another. A single central lock file allows all transitive dependencies to have locked versions defined at the root, thereby eliminating third-party dependency management software such as Paket. ## Explanation @@ -64,9 +64,9 @@ Some alternatives exist. Namely: The primary example of prior art for a centralized lock file is [Paket](https://fsprojects.github.io/Paket/) built around this exact concept. It uses a `paket.dependencies` file, similar to the `Directory.packages.props` file from Nuget. It allows for 'groups' to pull two identical dependencies in different versions as each group is resolved independently. It then uses a `paket.lock` to lock the dependencies at the root (right next to the paket.dependencies), which pins all primary and transitive package versions for all projects. A project then includes a package by listing its name in the `paket.references` file in the csproj directory. -We can draw many parallels between Paket & Nuget (with a central lock file) function. For one, we define all the primary packages & versions at the root. Projects define which specific packages they depend on within the relative projects. Where paket would use groups to include dependencies at different versions, NuGet uses the `VersionOverride` property within the csproj. This is the primary difference in how locks are handled for version differences since Paket utilizes groups and writes the locked package versions into the `paket.lock` file. Nuget (with central locking) writes all the pinned versions into the local csproj lock file. +We can draw many parallels between Paket & NuGet (with a central lock file) function. For one, we define all the primary packages & versions at the root. Projects define which specific packages they depend on within the relative projects. Where paket would use groups to include dependencies at different versions, NuGet uses the `VersionOverride` property within the csproj. This is the primary difference in how locks are handled for version differences since Paket utilizes groups and writes the locked package versions into the `paket.lock` file. Nuget (with central locking) writes all the pinned versions into the local csproj lock file. -NodeJS has the concept of [Workspaces](https://docs.npmjs.com/cli/v7/using-npm/workspaces#skip-to-content) which act more like Solutions since NodeJs only has the concept of Projects. However a Workspace does allow for all packages to be resolved to the same version, ensuring no version conflicts between different projects within the same workspace. +NodeJS has the concept of [Workspaces](https://docs.npmjs.com/cli/v7/using-npm/workspaces#skip-to-content) which act more like Solutions since NodeJS only has the concept of Projects. However a Workspace does allow for all packages to be resolved to the same version, ensuring no version conflicts between different projects within the same workspace. ## Unresolved Questions @@ -77,7 +77,7 @@ NodeJS has the concept of [Workspaces](https://docs.npmjs.com/cli/v7/using-npm/w * How would the `Directory.packages.props` dependencies be separated out from the project dependencies to generate the lock file? * How does the publish command handle this as it doesn't have full context of all packages. (Could be solved by the restore always running with --locked-mode) * How will a project deleted from the repository get deleted from the lock file. -* How will Lock files be generated with nested `Directoy.packages.props` files. +* How will Lock files be generated with nested `Directory.packages.props` files? ## Future Possibilities From f6d4ae78a7e3d51599265afb612efe2fbc84d24e Mon Sep 17 00:00:00 2001 From: CEbbinghaus Date: Wed, 20 Mar 2024 13:58:38 +1100 Subject: [PATCH 4/4] Added Additional benefit of TFM Lock file limitations --- accepted/2024/repo-level-lock-file.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/accepted/2024/repo-level-lock-file.md b/accepted/2024/repo-level-lock-file.md index fc306c68a..fa7e6d30d 100644 --- a/accepted/2024/repo-level-lock-file.md +++ b/accepted/2024/repo-level-lock-file.md @@ -81,4 +81,4 @@ NodeJS has the concept of [Workspaces](https://docs.npmjs.com/cli/v7/using-npm/w ## Future Possibilities -Using the `TargetFrameworks` property within the `Directory.packages.props` file could narrow down the frameworks resolved for the provided packages. This can simplify the lock file and reduce some unnecessary dependency resolution. +Using the `TargetFrameworks` property within the `Directory.packages.props` file could narrow down the frameworks resolved for the provided packages. This can simplify the lock file and reduce some unnecessary dependency resolution as well as reduce merge conflicts as there is less content changing less often