Toml dependencies

[halexiev-hedgeserv]

Hello,

Just wanted to ask what is your stance on toml dependencies and if you have considered using them in your project as an idiomatic way of describing them?

Kind regards,
Hristo

[gildor]

+1 here

Would be great to update project. I can try to contribute if you think it makes sense
In general it's a great project, would be great to keep it up to date

[jjohannes]

Here are my thoughts on the Version Catalog. Took a bit longer, because I still do not have enough experience with it myself to express this in a more compact way.

First, some context to relate Catalogs to other Gradle features:

• The Catalog is a a convenience on top of Core Gradle Features. It does not add anything new to Gradle's dependency management engine. Compared to e.g. dependency constraints or platforms, which were a substantial addition to make certain use cases work at all. The Catalog is built on top of all these things without adding/changing anything in the dependency resolution engine. This is not bad. It's good to have conventions and best practices on top of the raw Gradle Core Features. I am just stating this fact for context.
• Compared to other conveniences, like Convention Plugins, I am not (yet?) convinced that Catalogs are a good practice for every Gradle project. I.e. for Convention Plugins, it is clear that it is the way forward to organise build logic in every build. For the Catalog, there certainly is a set of projects where it is a good choice (I do not know though what the characteristics of these projects are). But I think there are also projects, where it makes sense to centralize versions by other means as long as you centralize them somehow - see my video on the topic. This is probably due to the nature of dependency management, where for many things there is no one-fits-all solution. It depends a lot on what kind of software your are building and which kind of dependencies you have and which versioning patterns are used.

With this in mind, here are the things thatI like and dislike about the Catalog approach. This are just my thoughts and maybe non of them are strong pro or con arguments. IMO, there are two aspects to look at separately (1) GA coordinates of a Component (2) Version of a component.

GA coordinates and aliasses

(pro) Tool support when accessing entries from the catalog in build scripts (code completion, only allow entries that exist). This is certainly a 'pro', although you can debate if certain tooling should be offered by the IDE directly. E.g. allowing code completion directly for coordinates from Maven Central. I think IDEA already does that for pom.xml files.

(pro or con) Aliases that add another indirection. This can be a pro, if you use a dependency in many places. So you can have a short alias for long things like com.fasterxml.jackson.datatype:jackson-datatype-jdk8 and don't have to repeat that several time. On the other hand, you then have yet to come up (and remember) another name for a thing. In situations where things are already complex because you have different coordinates for the same thing to deal with (think javax... vs jakarata...) it might just complicate things more. This is what I mean when I say that it depends on the kind of project (and on the taste and experience of the developers working with it).

(pro or con) TOML file. It's a pro for those who already managed versions in properties files before and like this format. On the other hand, it introduces yet another language for Gradle configuration, while Gradle already has the extensible DSLs that can be used to express everything. I don't like that there are complex notations like commons-lang3 = { group = "org.apache.commons", name = "commons-lang3", version = { strictly = "[3.8, 4.0[", prefer="3.9" } } in the TOML file. This screams to me that this should be done in Gradle's DSL. I tend to think, that for new users it's rather unnecessary complex that there is yet another file to use/learn about if the problem this attempts to solve can already be solved with Gradle's DSL.

Versions

What I love about Gradle's dependency constraints is that it makes this nice separation of the actual Component (addressed by GA coordinates) and its version. I think this makes a lot of things clearer and also easier to understand. So I like to see dependency declarations that do not include a version number. And define the whole structure of my software project in terms of dependency definitions between components. Without thinking about versions at all.
And then, I do the version management completely separated. Which works pretty well with a platform project.
I think the introduction of dependency constraints, that enable this, was a huge step forward.

Here is an example of what I mean from my gradle-project-setup-howto project:

A component's builld.gradle.kts

dependencies {
    api(project(":coruscant"))
    api("org.jboss.resteasy:resteasy-core")

    implementation("org.jboss.resteasy:resteasy-guice")
    implementation("org.jboss.resteasy:resteasy-jackson2-provider")

    testImplementation("org.junit.jupiter:junit-jupiter-api")
}

The platform for version management

dependencies {
    api(platform("com.fasterxml.jackson:jackson-bom:2.13.2.20220328"))
    api(platform("com.google.inject:guice-bom:5.1.0"))
}

dependencies.constraints {
    api("com.github.racc:typesafeconfig-guice:0.1.0")
    api("com.sun.activation:jakarta.activation:1.2.2") { version { reject("[2.0.0,)") } } 
    api("com.sun.mail:jakarta.mail:1.6.7") { version { reject("[2.0.0,)") } }
    api("jakarta.inject:jakarta.inject-api:1.0.5") { version { reject("[2.0.0,)") } } 
    api("jakarta.servlet:jakarta.servlet-api:4.0.4") { version { reject("[5.0.0,)") } } 
    api("junit:junit:4.13.2")
    api("org.apache.solr:solr-solrj:7.7.3") { version { reject("[8.0.0,)") } }
    api("org.apache.velocity:velocity-engine-core:2.3")
    api("org.apache.zookeeper:zookeeper:3.8.0")
    api("org.assertj:assertj-core:3.22.0")
    api("org.opensaml:opensaml:2.6.4")
    api("org.reflections:reflections:0.9.11") { version { reject("[0.9.12,)") } }
}

I also like to use BOMs when they exist for aligning versions of multi-component libraries (see link to full example above) and not define a version for the single components at all.

I am sorry 😄 , but I just love 😍 how clean and direct this is. I just feels right to me. I love it.

The Catalog breaks this again and thus feels like a step backwards to me. Now you again define the GAV together and then add - through the alias - the version to each build script directly. You also can't use BOMs that directly anymore. Of course you can also have BOMs in the catalog, but you don't have your own platform to wrap them nicely into one. You can do it in the catalog via bundles. But it feels a bit off to me that there is this new concept "bundle" - it somehow feels like half re-inventing the wheel.
But This just my feeling which I can't shake. In practice this "step backward" is hidden. Because in the build scripts, I only use the aliases and I don't actually repeat a version anywhere. So from a high level user perspective, my argument may not count at all.

Another thing I am unsure about is how the catalog works together with convention plugins, or is shared when you split your build into multiple included builds. With a platform, it's all so clear because the platform is a component itself that you can depend on everywhere where your need it with Gradle's dependency management. For the catalog, you have to reuse a plain TOML file in several places. It just feels more rough (and thus again like going backwards) to me. (But again maybe it is more a theoretical problem than a practical one.)

That's why my current conclusion is that it depends (on many things) if you should use the catalog or not.

You may notice, that I have trouble placing the Catalog in the overall Gradle "Model". With other concepts it's much clearer to me why they are there and where Gradle is heading with them. The catalog feels more like a "nice thing on top" that you may or may not use and which is also not connected to anything else.

This is a lot of text. I was not able to express this in a more compact way. 🤷 Hope it helps anyway.

@gildor PRs are always welcome. And if only to discuss. I think in this project, introducing the catalog can make sense. Especially because it has the ExternalLibrariesExtension right now. Which is like a self-built catalog for only GA coordinates - adding only the alias/code-completion part without breaking the GA vs Version separation. I am not sure if I would have added that if I would do this project today (I would probably just have used the coordinates directly everywhere as in the sample further up).
Maybe we can do it like this, that I maintain two branches (or even three) in this project with the different solutions.

[melix]

As the (main) author of version catalogs, I'd like to add more on the topic. First of all, I have deep experience on using them, since I have designed them, but also used them in many projects. There are intensively used at scale in the Micronaut projects, where they allow us a number of things:

• centralize dependency versions
• get automatic updates via Renovatebot
• standardize on dependency declaration notations so that we can use a single source of truth for both library declarations and BOM generation (each project publishes a BOM which is derived from the catalog)
• each of the project publishes a BOM and a catalog, which makes it extremely convenient to use for users (you don't have to think about dependency coordinates, you just hit implementation libs.mn.micronaut.data+CTRL+space and get completions about what you can use. For example this is extremely elegant, from my point of view: https://github.com/micronaut-projects/micronaut-test-resources/blob/4cb28b1944c254cc368989b4a7ce7d0a76a295ee/test-resources-hivemq/build.gradle#L12-L13

I didn't have to figure out the dependencies, I just had to import the Micronaut catalog and there you go.

The build scripts are so much cleaner. It also removes a lot of duplication, with many dependency strings replaced with a single, reusable identifier (the alias).

Note that in Micronaut, we're using single version dependencies, that is to say the traditional "require" statement. We don't use richer versions (yet) because it's hard to make it work for both Gradle and Maven consumers. That said, our update mechanism supports them too.

The other important part is that a catalog is not a platform, as I tried to describe in the docs. A catalog is, as the name implies, a list of libraries you can pick from, but they have no influence on the dependency graph. This is intentional, and for a framework like Micronaut it makes a lot of sense: there are many dependencies you may add to your graph, and if you want to add them, you have them at hand. Because we also publish a BOM, you get the best of the 2 worlds: something which can influence the dependency graph and single place where to define versions.

Last but not least, catalogs do not "pollute" publishing by having a dependency on a platform, which, often, is not what you want. It may be the case, but in general, I am tempted to say it's not. Moreoever there are thousand places where you can define the platform (submodule, buildSrc, included builds) vs single place for the catalog. But, again, you should probably have both.

Last but not least, catalogs are easy. They are easy to:

• write (TOML file FTW)
• read (even if you don't know Gradle, you read the file, you understand what is is about)
• update (tooling like renovatebot supports them)
• understand (no fancy concept of a platform to digest, not need to understand the difference with a BOM)
• integrate (it's a breeze to write a plugin on top of them which supports additional conventions based on the name of aliases, like we do in Micronaut for the BOM, in the catalog).

[jjohannes]

Thank you for the addition detailed thoughts Cédric.

My conclusion on this, for now, is that it depends on what you are building and how your overall dependency management setup for that is.

For this example, I have now added two commits on separate branches that show how to use the Version Catalog instead of the original setup. It‘s linked/documented in the Readme: https://github.com/jjohannes/idiomatic-gradle#summer-2022-update-gradle-751

[TWiStErRob]

@jjohannes I think it would be beneficial to swap out the default with version catalog and leave the custom (more advanced) versioning scheme as a branch, because people are referencing this repository and they're seeing that Version Catalogs are not idiomatic.

[jjohannes]

Having a custom extension with constants for the GA coordinates (which was the state on main) is indeed outdated now I would say as you can better use the Catalog then.

I changed the repo structure so that the catalog is now used on the main branch.

There are these alternative branches which might be preferable depending on how you want to do version management in your project (see all the discussions above):

• Do not use the Catalog - Removes the libs.versions.toml catalog and instead uses GA coordinates directly (versions are managed in platform project)
• Use only the Catalog (remove Platform) - Remove the gradle/platform project and put the versions directly into the catalog instead.

This is also documented in the Readme.

[jjohannes]

Something I am still wondering about is how to best share the Catalog in a multi-build (composite build) setup like this. This works:

dependencyResolutionManagement {
    versionCatalogs.create("libs") {
        from(files("../libs.versions.toml"))
    }
}

But it needs to be in quite some places. Maybe sharing the Catalog through dependency management (as you do for the platform) would be cleaner. And then you can seamlessly switch between a in-repository and published catalog (similar as for the platform).

This does not work without flaws as the catalog is required early in the build configuration. But you can work around this as we discussed here: gradle/gradle#19288 (comment)

[anderso]

But sharing it through dependency management does not solve the issue of repeating the dependencyResolutionManagement block in every project that uses it, right? It just changes it from referencing a file to a GAV instead:

dependencyResolutionManagement {
    versionCatalogs.create("libs") {
        from("group:id:version")
    }
}

I tried to solve this in my composite build by using a settings plugin, but then came across the issue (I suspect) that settings plugins don't quite work in a composite build. So even if the settings plugin resides in a project that is part of the composite build, it will not resolve the settings plugin unless I explicitly include the build in each project that uses the version catalog with this construct:

pluginManagement {
    includeBuild("...")
}

But then I can no longer build the project independently.

[jjohannes]

Catalog Sharing

I think being able to refer to the catalog by coordinates is exactly the advantage. Because these are always the same and you do not need to use a relative path that might not work anymore if you move something around.

This is what I would do: ed0545b

Settings Plugins (imo it's not directly related)

but then came across the issue (I suspect) that settings plugins don't quite work in a composite build.

Not sure what you mean. They work as intended (since Gradle 7.x). It's true that you need to explicitly include the settings plugin in each build. That's by design. So that each build also works individually. (And also because technically it's hard to solve the ordering - settings plugin needs to be found first.)

But then the settings plugin can be the only build you include and the only plugin you apply in all of your builds. Everything else can be controlled starting from there. It's done in this repo if you look at the builds in the /product folder.

[anderso]

I think being able to refer to the catalog by coordinates is exactly the advantage. Because these are always the same and you do not need to use a relative path that might not work anymore if you move something around.

Yes, I see now what you mean.

but then came across the issue (I suspect) that settings plugins don't quite work in a composite build.

Not sure what you mean. They work as intended (since Gradle 7.x). It's true that you need to explicitly include the settings plugin in each build. That's by design. So that each build also works individually. (And also because technically it's hard to solve the ordering - settings plugin needs to be found first.)

My comment was based on my expectation that gradle would find the settings plugin if it was part of the same composite build, not necessarily included from the same project that applies the plugin. So that I can have a project A that does not include any other build that uses the settings plugin resolved from my maven repository. But also be able to create a separate composite build B that includes both project A and the project containing the settings plugin, in which case the plugin would be resolved from the composite build. This works for build plugins (and of course regular dependencies) but not for settings plugins. But I guess this is not possible. And in any case it is not related to this issue.