A Proposal to Manage Dependency Updates

[ryanmoran]

The current set of dependencies delivered in each of the Paketo buildpacks is managed and updated by an external system which makes this process opaque to external users and complicated to modify as maintainers and contributors. To address some of these concerns, we should introduce tooling that can be consumed in our public GitHub workflows to perform these same operations.

We currently use jam to perform similar types of operations and so it might be a good place for this new functionality. Let's explore what that might look like:

A Set of Rules

Looking at the current set of dependencies we ship and the system that maintains them, it seems clear that we can express what dependencies should be included in a buildpack using a set of rules with three parts:

1. The ID of the dependency
2. A constraint that specifies the "version line" to be tracked
3. The number of patch release versions to keep for the given rule

Taking an example, we can express the versions that should be included in the bundler buildpack using the following rules:

id	constraint	patches
bundler	1.*	2
bundler	2.*	2

We can see that we track the 1.* and 2.* version lines of the bundler dependency and that we keep 2 patch versions of each line.

In continuing to review the rest of the dependencies included in the Paketo buildpacks, we can see that all of the current dependencies can be expressed with a similar set of rules.

id	constraint	patches
bundler	1.*	2
bundler	2.*	2
composer	*	2
dep	0.5.4	0
dotnet-aspnetcore	2.1.*	2
dotnet-aspnetcore	3.1.*	2
dotnet-aspnetcore	5.0.*	2
dotnet-runtime	2.1.*	2
dotnet-runtime	3.1.*	2
dotnet-runtime	5.0.*	2
dotnet-sdk	>=2.1.800 <2.1.900	100
dotnet-sdk	>=3.1.400 <3.1.500	100
dotnet-sdk	>=5.0.100 <5.0.200	100
dotnet-sdk	>=5.0.200 <5.0.300	100
go	1.14.*	2
go	1.15.*	2
go	1.16.*	2
httpd	*	2
icu	*	2
miniconda37	*	2
miniconda38	*	2
nginx	1.18.*	2
nginx	1.19.*	2
node	10.*	2
node	12.*	2
node	14.*	2
node	15.*	2
php	7.3.*	2
php	7.4.*	2
php	8.0.*	2
pip	*	2
pipenv	*	2
python	3.6.*	2
python	3.7.*	2
python	3.8.*	2
python	3.9.*	2
ruby	2.5.*	2
ruby	2.6.*	2
ruby	2.7.*	2
ruby	3.0.*	2
rust	1.*	2
tini	0.*	1
yarn	*	2

A Simple Algorithm

Given this set of rules we can construct an algorithm for finding the set of dependencies that should be included with any given buildpack.

1. Fetch all dependencies for the given ID: curl https://api.deps.paketo.io/v1/dependency?name=<id>.
2. Find all dependencies matching the version constraint.
3. Pick the latest N patch versions.

This algorithm can be repeated with each rule such that a complete set of dependencies matching all rules can be collected.

A Concrete Implementation

Given an algorithm to find those dependencies, we can imagine a more concrete implementation that involves some of our existing tooling and modifications to our configuration metadata. First, we can add some configuration to the existing buildpack.toml under the metadata field. Specifically, we can express the rules we want to use when including dependencies for the buildpack. Let's take a look at what that could look like with our bundler example from above:

[[metadata.dependency-constraints]]
  id = "bundler"
  constraint = "1.*"
  patches = 2

[[metadata.dependency-constraints]]
  id = "bundler"
  constraint = "2.*"
  patches = 2

As we can see, the rules from above are translated into TOML configuration data. We have a list of rules that outline what versions of the bundler dependency should be included in the buildpack. Then, given this configuration, we can add a new command to the existing jam CLI to allow our automation to update the dependencies. Something like the following might do the trick.

jam update-dependencies --buildpack-file buildpack.toml

Given this command, jam would read in the buildpack.toml file, find the rules, and then perform the algorithm outlined above to find the set of dependencies to include. Once that is done, it would replace the existing set of dependencies with the new set, thus keeping the dependencies up-to-date with the latest offered dependencies in the dep-server.

[thitch97]

@ryanmoran looking toward the future, is the idea that maintainers will need to manually update the dependency-constraints whenever new version lines emerge?

[ryanmoran]

@thitch97 Yes. I think this is ideal because it requires an acknowledgement by the buildpack maintainers that a new version line is now supported. At this point, they can consider if changes to the buildpack need to be made or if we need new test coverage.

[arjun024]

Really like the proposal. Couple of thoughts.

1. Probably have an --api <uri> option for the command.

2. A little concerned about the maintenance burden of having to know when a new release/constraint is available.
   Maybe also have a jam update-dependencies-constraints --buildpack-file buildpack.toml --api <uri> that updates the metadata from the dep server?
   This could be possibly used in a cron task that would update the metadata.dependency-constraints and open
   a Pull Request. Maintainers can manually review it, add tests etc. and merge.
   update-dependencies-constraints should ignore all versions older than the lowest constraint if exists.

[ryanmoran]

1. Probably have an --api <uri> option for the command.

This sounds totally reasonable and would allow others to use their own dep-server.

2. A little concerned about the maintenance burden of having to know when a new release/constraint is available.
   Maybe also have a jam update-dependencies-constraints --buildpack-file buildpack.toml --api <uri> that updates the metadata from the dep server?
   This could be possibly used in a cron task that would update the metadata.dependency-constraints and open
   a Pull Request. Maintainers can manually review it, add tests etc. and merge.
   update-dependencies-constraints should ignore all versions older than the lowest constraint if exists.

I think this introduces another set of rules that would need to be reverse-engineered and defined. Some of the dependencies keep major lines, others keep minor lines, still others keep only ranges of patch lines. This differs widely enough that you would need some set of rules that knows how to define the set of rules. It also brings back some of the concerns that @thitch97 has been looking into. How do we ensure that we have good coverage of a new line of dependencies if the whole process is automated?

[sophiewigmore]

@ForestEckhardt and I are spinning out separate issues to break this work down into smaller pieces:

• Backfill old dependency versions: Add missing dependency versions in use by buildpacks dep-server#26
• Jam update-dependencies: Add update-dependencies command to jam #178
• Add dependency constraint metadata to node-engine buildpack.toml: Add dependency constraint metadata to buildpack.toml node-engine#286
• Add dependency constraint metadata to yarn buildpack.toml: Add dependency constraint metadata to buildpack.toml yarn#98
• Create github workflow to automate dependency updates: https://github.com/paketo-buildpacks/github-config/issues/279\
• File issues across our buildpacks to switch over to the dep-server from the old dependency pipeline once we have seen this work in node-engine and yarn.
• Unblock Create automation for picking up new version lines from the dep-server github-config#296 and label as possible priority when this issue is completed

[ryanmoran]

There are some remaining automation issues to work out, but this work as required in packit is done. I'm going to close this out.