Uploading and releasing a multi-project build

[aalmiray]

This is perhaps more of a question than an actual bug/feature.

What's the minimal configuration required to publish a set of projects (the set may be a subset of all projects in the build) to Sonatype Nexus and release the staging repository once?

As I understand correctly, following the naming conventions in tasks, the plugin may create a staging repository per project. If that's the case, then multiple staging repositories would have to be closed & released, isn't it?

Thus, if it's possible to release a subset to the same staging repository so that a single staging repository is closed & released, I'd like to know how it may be done :-)

If it's not possible to have this scenario with the current release, then please consider adding it. As a reference the Bintray plugin registers a task at the root project that depends on individual bintrayUpload tasks (one per subproject). This "aggregating" task would then wait til all subtasks finish then proceed to sync uploaded artifacts to Maven Central. If not done already then this plugin would require a similar aggregating capability to close & release a single staging repository.

[szpak]

First of all, publishing (uploading) and closing/releasing of a multi-project build is supported out of the box, assuming that you want to do it for all subprojects (e.g. different module of one project, not a mono repo with separate components). See our e2e test project.

Nevertheless, close and release tasks are only orchestrated to mustRunAfter the publishing tasks. Therefore, in a one call you should be able to call only those publishing tasks that you want to (e.g. 2 of 5 suprojects) followed with the close/release tasks (just one staging repository should be explicitly initialized just before the first publishing task is called and reused by the others). Would it be enough in your case?

Please note, however, up until #19 is implemented you need to do it (publishing and closing) in a one Gradle call.

[aalmiray]

Well, that's a lot of ... Kotlin code. If I understood that example correctly, the nexus plugin is only applied to the root while the subprojects configure themselves with java-library, maven-publish, and sign plugins. If this is correct them all projects will be published to a staging repository from which closing & release may occur.

How then must a subproject be marked to be skipped from uploading to a staging repository?

[vlsi]

How then must a subproject be marked to be skipped from uploading to a staging repository?

Well, Gradle does whatever you ask it to do. If you ask it to publish all the artifacts, it does so. If you ask it to execute a subset of tasks it agrees, and it does not insist on doing all the tasks for all the projects every time 😉
That is the default behavior and it is not related to publish-plugin.

Could you clarify what do you mean by "marked to be skipped"?

What the plugin does is as follows: it adds a task to the root project to initialized the staging repository, then it rewrites all the URLs for all projects to point to the newly created staging repository. The singing and publishing are out of scope for gradle-nexus/publish-plugin.

[szpak]

Well, that's a lot of ... Kotlin code.

Some people prefers that over Maven's XML ;-). We have also a complementary variant in Groovy, but just for a single module.

As @vlsi already suggested, try to just call :project1:publishToSonatype :project2:publishToSonatype closeStagingRepository.

[aalmiray]

Good grief, what if I have more than 50+ projects (as Ikonli does). Do you expect people to:

a) manually invoke :X:publishToSonatype where x = the number of projects?
b) create a task at the root level that collects al projects that should be published to Sonatype?

Again, the reason why I ask this is that the Bintray plugin handled aggregation automatically. I get it that this plugin provides the minimum building blocks to get the job done, I'm asking for convenient options that can be turned on/off when necessary. What's the point of having convention over configuration if not to avoid defining stuff over and over again?

[aalmiray]

How then must a subproject be marked to be skipped from uploading to a staging repository?

Well, Gradle does whatever you ask it to do. If you ask it to publish all the artifacts, it does so. If you ask it to execute a subset of tasks it agrees, and it does not insist on doing all the tasks for all the projects every time 😉
That is the default behavior and it is not related to publish-plugin.

I'm well aware of that.

Could you clarify what do you mean by "marked to be skipped"?

gradle publish will invoke the publish task on all matching projects. This plugin adds publishToSonatype to all matching plugins. What I want is to avoid adding publishToSonatype (and any other tasks brought by this plugin) to a subproject (or at least make then disable by default without me explicitly saying so on each individual task (which I know can be done)).

What the plugin does is as follows: it adds a task to the root project to initialized the staging repository, then it rewrites all the URLs for all projects to point to the newly created staging repository. The singing and publishing are out of scope for gradle-nexus/publish-plugin.

I'm well aware of that.

[szpak]

@aalmiray I'm afraid, I do not fully follow your case.

Do you have a multiproject build with
a) let's say 9 publishable subprojects, but you want to be able to publish 4 of them (a fixed set) some days and 5 others (the other fixed set) some other days?
b) 9 subprojects at in general, where 3 of them are publishable (and you just want to release them to Maven Central) and 6 others are just test modules or some other auxiliary stuff that cannot be published at all?
c) anything else :-)?

[aalmiray]

That would be case b). Not all projects are publishable (for whatever reason) hence why I used the word subset in the beginning.

[aalmiray]

Add up that some subprojects are publishable, just not to Maven Central. Could be to other Maven compatible repositories (non Nexus managed), could be a Gradle plugin (which would meet the criteria for publication but to a totally different repository).

So in essence, I'd like to target just a subset of publishable subprojects with little to no configuration (conventions, let's use them) and without being forced to add custom code in build scripts (or pre compiled plugin scripts) or specifying publish targets in the command line.

[szpak]

That would be case b). Not all projects are publishable (for whatever reason) hence why I used the word subset in the beginning.

In that case, I assume there is no maven-publish plugin applied in those other projects, right? Then you can just call:

./gradlew publishToSonatype closeStagingRepository

and the plugin should automatically detect "publishable" subprojects, initialize the staging repository before the first artifacts upload and close/release that staging repository after the last artifacts are uploaded.

[szpak]

Add up that some subprojects are publishable, just not to Maven Central. Could be to other Maven compatible repositories (non Nexus managed), could be a Gradle plugin (which would meet the criteria for publication but to a totally different repository).

It's somehow more complicated. We don't have that mechanism out of the box. Probably, it will be implemented once better support for monorepos is requeted (and provided). At the time being, you could try to use your own -PreleaseToCentral flag and ignore/skip publish tasks in the subprojects that should not be published to Maven Central (in those tasks configuration).

Update. If you have a good idea how we could distinguish the projects you would like to skip (without their enumeration in the project configuration or some marker property set) please let us know.

[aalmiray]

It's somehow more complicated. We don't have that mechanism out of the box. Probably, it will be implemented once better support for monorepos is requeted (and provided). At the time being, you could try to use your own -PreleaseToCentral flag and ignore/skip publish tasks in the subprojects that should not be published to Maven Central (in those tasks configuration).

I don't see what monorepo support has to do with this question. This is for a typical multi-project build where all subprojects reside in the same repository.

Update. If you have a good idea how we could distinguish the projects you would like to skip (without their enumeration in the project configuration or some marker property set) please let us know.

Given that the plugin already configures all projects in the build, using the extension it provides could be the way to go. Add a boolean property to mark the project for inclusion (default = true).

[szpak]

I don't see what monorepo support has to do with this question.

In monorepos there is (might be) a need to release - on demand - only 3 projects (for example sharing some "subroot" or not) from 100, available in the whole repo, publishable projects. So, some kind of the mechanism we are talking about would problably be required.

Given that the plugin already configures all projects in the build, using the extension it provides could be the way to go. Add a boolean property to mark the project for inclusion (default = true).

We need to collect publishable projects in one place and synchronize the whole operation (init repo before the first upload and close/release after last). As a result this plugin is applied only in the root project and it tries to find all publishable (sub)projects. Therefore, there is no extension created in subprojects.

We could try to check for some value in the ExtraPropertyExtension (not very elegant, but probably easy to implement) or provide a way to allow people define their own logic to find desired projects.

[joebowbeer]

I'm realizing that nexusPublishing needs to be defined at the root, and therefore group and version must also be defined at the root. This is causing a lot of disruption in my Android project, where only the library subproject is published.

[jvmlet]

@aalmiray , I think you can exclude sub-module from being published to nexus by NOT applying maven-publish plugin, this also can be done dynamically in gradle script :

->sub-project-a
    buld.gradle :
    apply plugin: 'maven-publish'
->sub-project-b
    buld.gradle :
      if (someCondition){apply plugin: 'maven-publish'}
->sub-project-c
    buld.gradle:
     // 'maven-publish' plugin is NOT applied
buidl.gradle:
plugins {
    id("io.github.gradle-nexus.publish-plugin") version "«version»"
}

Then invoking ./gradlew publishToSonatype closeStagingRepository will publish sub-project-a and sub-project-b only if someCondition evaluates to true

@szpak , can you please confirm this ?

[aalmiray]

Yes, but no. I do want the maven-publish plugin for other reasons. I simply do not wish a specific subset of modules to be published to Maven Central. current the GNPP performs a blanket set of values on every module.

[marcphilipp]

Here's a workaround for disabling the corresponding publishing tasks for the projects that should not be published to Maven Central in the build script:

nexusPublishing {
	repositories {
		sonatype()
	}
}
val nonMavenCentralProjects = listOf(
	project(":a"),
	project(":b")
)
nonMavenCentralProjects.forEach { subproject ->
	subproject.tasks.withType<PublishToMavenRepository>().configureEach {
		onlyIf {
			repository.name != "sonatype"
		}
	}
}

I agree, though, that it would be better to not create the publishing repository for those projects in the first place. I can see us adding an API to configure includes/excludes, potentially as a predicate, e.g.:

nexusPublishing {
	includeProjects(project(":a"), project(":b"))
	includeProjects { it.name.startsWith("some-prefix") }
	excludeProjects(project(":c"), project(":d"))
	excludeProjects { it.name.startsWith("some-other-prefix") }
}

@aalmiray Whichone would be better for your use case?

@szpak WDYT?

[aalmiray]

I'd prefer if the include/exclude lists were provided by the plugin's extension. Less code to copy around by consumers IMHO.

[marcphilipp]

So sth. like this?

nexusPublishing {
	includedProjects.set(mavenCentralProjects)
}

[aalmiray]

It depends. What is mavenCentralProjects?

If it's a list of String or Project then the value must be calculated eagerly.
If it's a predicate then the list is calculated on demand when applied to rootProject and navigating all children projects.

Given the preference to defer computations as much as you can to speed up the configuration phase I'd say the predicate option would be best. You could also accept both options for those that prefer eager evaluation for whatever reason they deem necessary.

And it has to be both include and exclude lists with the caveat that when both lists are defined the exclusions are taken from the inclusions.

[aalmiray]

Ping. Wondering if the team has circled back to @marcphilipp's latest idea on this topic.

[vlsi]

I wonder if requiring Gradle 6.2 and moving to Shared Build Services would help here.
Then we no longer need to use "root project" as a holder for the staging repository and things like that.

[vlsi]

if the include/exclude lists were provided by the plugin's extension

The current inclusion rule is the presence of maven-publish plugin. If you apply the plugin to the publishable projects only, then it would act as a filter.

publish-plugin/src/main/kotlin/io/github/gradlenexus/publishplugin/NexusPublishPlugin.kt

Line 129 in 1c7b509

plugins.withId("maven-publish") {

As a better solution, I would suggest moving that "withId maven-publish" into another sub-plugin.

In other words:
gnpp-base plugin: creates build service, and the init/close tasks
gnpp-publish plugin: creates tasks in for publishing of a given project

gnpp plugin (the current one): applies gnpp-base, then applies gnpp-publish to allprojects.
The ones who need customization would be able to apply gnpp-base and gnpp-publish as they see fit

[aalmiray]

I understand the current conditional on maven-publish and that doesn't work for my use case as that plugin is applied to all subprojects by default. Would your suggestion to split GNPP in two help in this regard?

[vlsi]

doesn't work for my use case as that plugin is applied to all subprojects by default

Would you please clarify (no trolling, really) why do you apply maven-publish for all the subprojects?
Do you think you could limit maven-publish to projects that really need to be published? (e.g. by adding a property
to the project, or by using a naming scheme).

For instance:

subprojects {
    if (!name.contains("-test-")) {
        apply(plugin = "maven-publish")
    }
}

Would your suggestion to split GNPP in two help in this regard?

I think the split of GNPP would move the behavior towards all the other Gradle plugins.
It would be really sad if many Gradle plugins had their own ways to configure include/exclude filters.

In other words, like with maven-publish, you do not really have "an extension that enables to configure include-exclude filters". You have options to apply the plugin or skip applying the plugin.

I do not say that all GNPP users have to suffer and copy-paste 100500 lines of configuration code. However, it looks like allprojects is a code smell in Gradle plugins since there might be cases where build users need to configure filtering.
And, it looks like factoring out the actions into a separate plugin helps with that: users can choose to apply or skip the relevant plugins (==relevant build logic bits).

That seems to be a reasonable approach, especially in the idiomatic gradle case when every build.gradle.kts declares all its plugins via plugins {...} section only.

[aalmiray]

The org.kordamp.gradle.publishing plugin applies the mavne-publish plugin to all projects in a particular multi-project setup. See https://github.com/kordamp/kordamp-gradle-plugins/blob/e744cb7197e8ae0d9cc66c4b064f50776f08723d/plugins/publishing-gradle-plugin/src/main/groovy/org/kordamp/gradle/plugin/publishing/PublishingPlugin.groovy#L66-L109

The idea is to let subprojects determine if publication should be enabled or disabled using the plugin's DSL, such as

config {
    publishing {
        enabled = false
    }
}

Because of the nature of the maven-publish plugin that configured Publications in an afterEvaluate block, the publishing plugin is currently configured to react when the DSL has been updated by the developer. Given how the plugin is currently designed (applied at the root then immediately applied to children projects) the filtering option you suggest using the project name won't work in this case.

An alternative would be to read a project property (instead of a DSL property) as this type of properties have their values visible immediately upon query, in contrast the plugin's DSL properties have not been signed a value by the developer as the plugin's apply() is currently being executed by the time it determines if maven-publish should be added or not. Using a project property for this case "works" but breaks the contract of having the configuration be specified by DSL properties alone.

Thsi is why I was interested in reading @marcphilipp's suggestion of using GNPP's existing extension (DSL) to achieve what I'm looking for.

[vlsi]

Because of the nature of the maven-publish plugin that configured Publications in an afterEvaluate block

This is a nice catch.
I believe afterEvaluate there is not needed, and it should probably be removed:

publish-plugin/src/main/kotlin/io/github/gradlenexus/publishplugin/NexusPublishPlugin.kt

Lines 125 to 129 in 1c7b509

	private fun configurePublishingForAllProjects(rootProject: Project, extension: NexusPublishExtension, registry: Provider<StagingRepositoryDescriptorRegistry>) {
	rootProject.afterEvaluate {
	allprojects {
	val publishingProject = this
	plugins.withId("maven-publish") {

On the other hand, root project is always evaluated first, so there should be no difference.
It does not mean "afterEvaluate of each project". It means "afterEvaluate of the root", which is always the first.

publishing plugin is currently configured to react when the DSL has been updated by the developer

I have no idea how to resolve that. Gradle DSL does not allow to "freeze configuration", and it provides no way to tell "this is fully configured, you might analyze it, create tasks, etc".

Frankly speaking, I am inclined that listing the plugin in plugins section is a good way to specify the developer wanted the feature, so "reacting to DSL usage" sounds like a code smell.

suggestion of using GNPP's existing extension (DSL) to achieve what I'm looking for.

If GNPP splits plugins, then it would be useful for everybody:

• the ones who prefer plugins {...} could apply gnpp-publish at the relevant level (gnpp-publish might even automatically apply maven-publish)
• kordamp might use it as well depending on its configuration: it would apply or skip gnpp-publish

Adding various include/exclude filters looks odd since it does not play well with plugins { ... }, it creates an unexpected configuration (all projects receive GNPP treatment, even the ones that did not really want that), it is sad to enumerate "published to central" in the root project, and so on.

That is why I am inclined that adding filtering to GNPP resolves a narrow use case (~ kordamp only?), it creates GNPP-specific filtering rules (the filtering rules would proliferate across various plugins), and it might create GNPP-specific issues like "I want to filter projects based on their property or ext variable, however, the ext is not seen in the root project". Users would bombard GNPP with questions like "how should I filter" which is hardly good for the maintainers.

[aalmiray]

Perhaps I explained myself badly. The core maven-publish plugin uses afterEvaluate to realize all publications, there is no workaround for it unless Gradle core changes. GNPP in turn could use a different mechanism than afterEvaluate if needed

There is a ticket at the Gradle issue tracker stating they want to remove apply plugins: in favor of plugins { ... } effectively removing the option to programmatically apply a condition to figure out if a plugin should be applied or not. IMHO this is a step backwards as it limits options and Gradle was supposed to be more flexible than its competitors. But hey, if the Gradle team wants this block gone then they'll get it.

And yes, so far it seems that adding includes/excludes to GNPP would benefit Kordamp directly but not so much other as no one else has brought up this concern before, have they? In which case I also wonder if indeed this should be added to GNPP or find another way in Kordamp to workaround this.

[vlsi]

The core maven-publish plugin uses afterEvaluate to realize all publications, there is no workaround for it unless Gradle core changes

Let me give it another try :)
Do you mean gradle/gradle@3800e74 change? (Gradle 5.0 makes publications {..} a regular block)

It looks like Gradle 5.0+ (2018-11-26) realizes publications eagerly: https://github.com/gradle/gradle/blob/39a81fc5e1eed33b98ef4e2561db54e08874fa13/subprojects/publish/src/main/java/org/gradle/api/publish/plugins/PublishingPlugin.java#L85

So I do not see which afterEvaluate do you mean.

no one else has brought up this concern before, have they?

#81 is somewhat related.

#109 is interesting as well since it wants "publishing multiple unrelated groups into a single staging repository".

I wonder what happens if nexusPublishing block was allowed in any project (not just root one), and then, at execution time, GNPP merged the publishing requests coming to the same Nexus and diverted them to the same staging repo.

AFAIK root project was used as a container for "staging repository id" earlier, however, Gradle 6.1 has shared build services, so the's no need in root project for GNPP.

effectively removing the option to programmatically apply a condition to figure out if a plugin should be applied or not

A similar case happens when plugins need to treat whenObjectRemoved case.
I wonder if would be a reasonable request (Gradle issue) for unapply(plugin = "...") or rollback(plugin = "...") which would effectively ask the plugin to undo all actions it has made (and disable all tasks it has registered).

WDYT?

[serpro69]

Hi,
Any progress on this issue or good workarounds?

I have exactly same situation - I have 3 submodules in a project, all of them have maven-publish applied, but I only want to publish 2 of them to nexus.

[aalmiray]

FWIW I no longer use this plugin for the required behavior. I use JReleaser + Kordamp instead