Add version catalog support for Gradle

[alcere]

Hello from the GitHub Android mobile team!

As part of a client apps hackathon we decided to give it a shot and add version catalog support for Gradle projects.

This functionality has been requested by some users here: #3121, and we are planning to start using version catalog as well as soon as Dependabot supports it :P.

We have added support for version catalogs defined using a toml file (more info), for libraries and plugins using value versions and referenced versions, so this changes add support for a toml file like:

[versions]
corektx = "1.7.0"
lifecycleRuntime = "2.3.1"
composeUi = "1.2.0"
jUnitVersion = "4.13.2"
androidxJunit = "1.1.3"
espresso = "3.5.0"

[libraries]
corektx = { module = "androidx.core:core-ktx", version.ref = "corektx" }
lifecycleRuntimeKtx = { module = "androidx.lifecycle:lifecycle-runtime-ktx", version.ref = "lifecycleRuntime" }

compose-activity = { module = "androidx.activity:activity-compose", version = "1.3.1" }
compose-ui = { module = "androidx.compose.ui:ui", version.ref = "composeUi" }
compose-ui-tooling = { module = "androidx.compose.ui:ui-tooling-preview", version.ref = "composeUi" }
compose-material = { module = "androidx.compose.material:material", version = { strictly = "[1.2.0, 1.3.1[", prefer="1.3.0" } }

test-junit = { module = "junit:junit", version.ref = "jUnitVersion" }
test-andoridx-junit = { module = "androidx.test.ext:junit", version.ref = "androidxJunit" }
test-compose-ui = { module = "androidx.compose.ui:ui-test-junit4", version.ref = "composeUi" }
test-compose-ui-tooling = { module = "androidx.compose.ui:ui-tooling", version.ref = "composeUi" }
test-compose-manifest = { module = "androidx.compose.ui:ui-test-manifest", version.ref = "composeUi" }
test-espresso-Core = { module = "androidx.test.espresso:espresso-core", version.ref = "espresso" }

[bundles]
compose = ["compose-activity", "compose-ui", "compose-ui-tooling", "compose-material"]

[plugins]
kotlinter = { id = "org.jmailen.kotlinter", version = "3.11.0" }

Testing

In order to guide our development we have used the steps defined in here to use the dry-run script pointing to this test repo we created.

Note

We realize that this code won't reach the quality bar to be considered as a candidate to be merged (Ruby is not our field of expertise), but we will be more than happy to react to your feedback and keep working on it!

BIG THANKS

To @jurre who has been helping us on this task, sharing advices, techniques and ideas!

Closes #3121.

[trevjonez]

As is this would cover the bulk of projects using version catalogs. But surely as things scale up and out we would see projects that have multiple catalogs or non conventional file layouts in use.

[jeffwidman]

👋 from the Dependabot team. Thanks for this great PR... it is definitely on our radar, although no one has had time to review it just yet. TBH, for myself it may be a few more weeks as I'm juggling a few things here at the beginning of the quarter and this is a larger change that will take a bit of time to review. But if no one else on the team gets to it first, I expect to be able to get to it within a few weeks.

[deivid-rodriguez]

This is pretty awesome! I made a first pass on reviewing.

My main concern is that when sharing version declarations between multiple dependencies, I think Dependabot may create a different PR for each dependency that uses the reference, all of them with the same contents? Is this correct?

[dsvensson]

Would it be more representable to look at the gradle lockfile? The libs.versions.toml file doesn't know about transitive dependencies (but maybe dependabot crawls that graph?), nor if these are updated via gradle version overrides.

[trevjonez]

Would it be more representable to look at the gradle lockfile? The libs.versions.toml file doesn't know about transitive dependencies (but maybe dependabot crawls that graph?), nor if these are updated via gradle version overrides.

FWIW: In the 8 years I have been using Gradle, I have never worked on a project that used Gradle lock files. I suspect this is more related to how mobile builds and deployments differ from a backend system CI/CD pipeline. Typically we run tools that scan and monitor the fully resolved tree before releasing. IE snyk.io

[dsvensson]

FWIW: In the 8 years I have been using Gradle, I have never worked on a project that used Gradle lock files. I suspect this is more related to how mobile builds and deployments differ from a backend system CI/CD pipeline. Typically we run tools that scan and monitor the fully resolved tree before releasing. IE snyk.io

This is because lockfiles was made available with 7.0 at 9 April 2021 so for 6 years and 2 months of those 8 years this feature wasn't available. Added to that, IDEA didn't really support it in the beginning, so maybe bump that to the first 7 of your 8 years. I know it's underused, but that doesn't change anything. It is the only way to avoid false positives, just wanted note that, could be a future improvement. Hopefully some future supply chain poisoning with characteristics that would make it preventable with lockfiles and dependency verification happens, so broad adoption happens.

[trevjonez]

FWIW: In the 8 years I have been using Gradle, I have never worked on a project that used Gradle lock files. I suspect this is more related to how mobile builds and deployments differ from a backend system CI/CD pipeline. Typically we run tools that scan and monitor the fully resolved tree before releasing. IE snyk.io

This is because lockfiles was made available with 7.0 at 9 April 2021 so for 6 years and 2 months of those 8 years this feature wasn't available. Added to that, IDEA didn't really support it in the beginning, so maybe bump that to the first 7 of your 8 years. I know it's underused, but that doesn't change anything. It is the only way to avoid false positives, just wanted note that, could be a future improvement. Hopefully some future supply chain poisoning with characteristics that would make it preventable with lockfiles and dependency verification happens, so broad adoption happens.

Was it really only so recently? Surprised but does explain it.
Makes me wonder the format difference between nebula and gradle's built in lockfile is.
https://github.com/nebula-plugins/gradle-dependency-lock-plugin

That one has been available for a long time, though likely hard to say how widely used it is outside of netflix.

[deivid-rodriguez]

I'll be working on this PR next week. Will push up my initial review suggestions, test out the code here a bit more, and in general try move this PR forward.

[alcere]

I'll be working on this PR next week. Will push up my initial review suggestions, test out the code here a bit more, and in general try move this PR forward.

@deivid-rodriguez sorry for taking so long to get you back 😓! I've been quite busy all this time. Do you mind if I address your comments? I have some spare time this week so I'd like to keep on working on this pull request.

[deivid-rodriguez]

@deivid-rodriguez sorry for taking so long to get you back 😓! I've been quite busy all this time. Do you mind if I address your comments? I have some spare time this week so I'd like to keep on working on this pull request.

No worries and great having you back in this PR :), thanks for addressing my comments. I see that the PR appears as out of date and a lot of old commits appear in its timeline, I think it needs a rebase?

Can you answer this question from my first review? I lost context now and I haven't yet got to reviewing this again but at the time I was not clear about it:

My main concern is that when sharing version declarations between multiple dependencies, I think Dependabot may create a different PR for each dependency that uses the reference, all of them with the same contents? Is this correct?

[alcere]

My main concern is that when sharing version declarations between multiple dependencies, I think Dependabot may create a different PR for each dependency that uses the reference, all of them with the same contents? Is this correct?

That's a great question @deivid-rodriguez , do you mean something like this?

[versions]
composeUi = "1.2.0"

[libraries]
compose-ui = { module = "androidx.compose.ui:ui", version.ref = "composeUi" }
compose-ui-tooling = { module = "androidx.compose.ui:ui-tooling-preview", version.ref = "composeUi" }
test-compose-ui = { module = "androidx.compose.ui:ui-test-junit4", version.ref = "composeUi" }
test-compose-ui-tooling = { module = "androidx.compose.ui:ui-tooling", version.ref = "composeUi" }
test-compose-manifest = { module = "androidx.compose.ui:ui-test-manifest", version.ref = "composeUi" }

[deivid-rodriguez]

Yes, I think that's what I meant.

[alcere]

Yes, I think that's what I meant.

You are very right, it does exactly that, I will look into the issue, thanks for the heads up!

[deivid-rodriguez]

@alcere I had another look at this today and I think this is actually fine. We already have logic in place in the Gradle ecosystem to be able to group PRs together when they are updating a "property" so I think this is ok as is! 🎉

I also reviewed the PR, cleaned up git history and fixed a few bugs while I was at it. Were you planning on pushing any more changes? If not, I can push my cleanup and I think this would be mostly ready! 💪

[alcere]

@alcere I had another look at this today and I think this is actually fine. We already have logic in place in the Gradle ecosystem to be able to group PRs together when they are updating a "property" so I think this is ok as is! 🎉

I also reviewed the PR, cleaned up git history and fixed a few bugs while I was at it. Were you planning on pushing any more changes? If not, I can push my cleanup and I think this would be mostly ready! 💪

That's great news! Please go ahead!

[deivid-rodriguez]

I pushed my changes. These include:

• Cleanup git history into a handful of meaningful commits (removed style amendment commits and things like that).
• Made some code simplifications and some style changes to make the implementation more "standard Ruby".
• I added support for dependency libraries specified as group:artifact:version or dependency plugins specified as id:version.
• I fixed an issue where Dependabot would crash when dependencies included the string "version" or the string "ref".

[yeikel]

If possible, let's please add a test case to cover this #6742

[deivid-rodriguez]

@yeikel The error you reported at #6742 was indeed still happening. I added a spec to make sure dependencies specified like that are ignored for now, and don't cause a crash in the update process like the one reported.