Skip to content

Commit a443217

Browse files
snazysnazy
committed
WIP Release infra stuff
Introduce a couple of shell scripts to automate the release process. Some docs are included under `releases/README.md`. Generally, releases at the ASF follow the following workflow: 1. Draft a release 2. Start a VOTE on the dev mailing list 3. If the VOTE fails, the release has failed - "go to step 1" 4. If the VOTE passes, publish the release The above process is, without release branches, reflected in the scripts: 1. `releases/bin/draft-release.sh --major <major-version-number> --minor <minor-version-number> --commit <Git-commit-ID>` 2. if the vote passes: `releases/bin/publish-release.sh --major <major-version-number> --minor <minor-version-number>` 3. if the vote fails, just run `draft-release.sh` again The change includes scripts to handle version branches, however, using those is not required for the two release scripts above. It's important to know that the scripts handle changes to the `version.txt` file and that a specific syntax for Git tags is expected, which is required to automatically use/generate RC and final versions, including the artifact publishing to Nexus.
1 parent 9b5325b commit a443217

20 files changed

+2480
-148
lines changed

‎NEXT_RELEASE_NOTES.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
19+
20+
Content to be copied into the next release notes, after the Contributors section.
21+
This file is maintained manually!
22+
-->

‎build-logic/src/main/kotlin/publishing/PublishingHelperPlugin.kt‎

Lines changed: 0 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -44,15 +44,8 @@ import org.gradle.plugins.signing.SigningPlugin
4444
* The task `sourceTarball` (available on the root project) generates a source tarball using `git
4545
* archive`.
4646
*
47-
* The task `releaseEmailTemplate` generates the release-vote email subject + body. Outputs on the
48-
* console and in the `build/distributions/` directory.
49-
*
5047
* Signing tip: If you want to use `gpg-agent`, set the `useGpgAgent` Gradle project property
5148
*
52-
* The following command publishes the project artifacts to your local maven repository, generates
53-
* the source tarball - and uses `gpg-agent` to sign all artifacts and the tarball. Note that this
54-
* requires a Git tag!
55-
*
5649
* ```
5750
* ./gradlew publishToMavenLocal sourceTarball -Prelease -PuseGpgAgent
5851
* ```

‎build-logic/src/main/kotlin/publishing/rootProject.kt‎

Lines changed: 12 additions & 130 deletions
Original file line numberDiff line numberDiff line change
@@ -19,14 +19,11 @@
1919

2020
package publishing
2121

22-
import io.github.gradlenexus.publishplugin.NexusPublishExtension
2322
import io.github.gradlenexus.publishplugin.NexusPublishPlugin
24-
import io.github.gradlenexus.publishplugin.internal.StagingRepositoryDescriptorRegistryBuildService
2523
import org.gradle.api.Project
26-
import org.gradle.api.services.BuildServiceRegistration
24+
import org.gradle.api.tasks.Delete
2725
import org.gradle.api.tasks.Exec
2826
import org.gradle.kotlin.dsl.apply
29-
import org.gradle.kotlin.dsl.named
3027
import org.gradle.kotlin.dsl.register
3128
import org.gradle.plugins.signing.Sign
3229

@@ -41,11 +38,22 @@ internal fun configureOnRootProject(project: Project) =
4138
val isRelease = project.hasProperty("release")
4239
val isSigning = isRelease || project.hasProperty("signArtifacts")
4340

41+
val cleanDistributionsDir = tasks.register<Delete>("cleanDistributionsDir")
42+
cleanDistributionsDir.configure {
43+
outputs.cacheIf { false }
44+
45+
val e = project.extensions.getByType(PublishingHelperExtension::class.java)
46+
delete(e.distributionDir)
47+
}
48+
4449
val sourceTarball = tasks.register<Exec>("sourceTarball")
4550
sourceTarball.configure {
4651
group = "build"
4752
description =
4853
"Generate a source tarball for a release to be uploaded to dist.apache.org/repos/dist"
54+
outputs.cacheIf { false }
55+
56+
dependsOn(cleanDistributionsDir)
4957

5058
val e = project.extensions.getByType(PublishingHelperExtension::class.java)
5159
doFirst { mkdir(e.distributionDir) }
@@ -85,130 +93,4 @@ internal fun configureOnRootProject(project: Project) =
8593
}
8694
sourceTarball.configure { finalizedBy(signSourceTarball) }
8795
}
88-
89-
val releaseEmailTemplate = tasks.register("releaseEmailTemplate")
90-
releaseEmailTemplate.configure {
91-
group = "publishing"
92-
description =
93-
"Generate release-vote email subject + body, including the staging repository URL, if run during the Maven release."
94-
95-
mustRunAfter("initializeApacheStagingRepository")
96-
97-
doFirst {
98-
val e = project.extensions.getByType(PublishingHelperExtension::class.java)
99-
val asfName = e.asfProjectId.get()
100-
101-
val gitInfo = MemoizedGitInfo.gitInfo(rootProject)
102-
val gitCommitId = gitInfo["Apache-Polaris-Build-Git-Head"]
103-
104-
val repos = project.extensions.getByType(NexusPublishExtension::class.java).repositories
105-
val repo = repos.iterator().next()
106-
107-
val stagingRepositoryUrlRegistryRegistration =
108-
gradle.sharedServices.registrations.named<
109-
BuildServiceRegistration<StagingRepositoryDescriptorRegistryBuildService, *>
110-
>(
111-
"stagingRepositoryUrlRegistry"
112-
)
113-
val staginRepoUrl =
114-
if (stagingRepositoryUrlRegistryRegistration.isPresent) {
115-
val stagingRepositoryUrlRegistryBuildServiceRegistration =
116-
stagingRepositoryUrlRegistryRegistration.get()
117-
val stagingRepositoryUrlRegistryService =
118-
stagingRepositoryUrlRegistryBuildServiceRegistration.getService()
119-
if (stagingRepositoryUrlRegistryService.isPresent) {
120-
val registry = stagingRepositoryUrlRegistryService.get().registry
121-
try {
122-
val stagingRepoDesc = registry.get(repo.name)
123-
val stagingRepoId = stagingRepoDesc.stagingRepositoryId
124-
"https://repository.apache.org/content/repositories/$stagingRepoId/"
125-
} catch (e: IllegalStateException) {
126-
"NO STAGING REPOSITORY ($e)"
127-
}
128-
} else {
129-
"NO STAGING REPOSITORY (no registry service) !!"
130-
}
131-
} else {
132-
"NO STAGING REPOSITORY (no build service) !!"
133-
}
134-
135-
val asfProjectName =
136-
e.overrideName.orElse(project.provider { "Apache ${fetchAsfProjectName(asfName)}" }).get()
137-
138-
val versionNoRc = version.toString().replace("-rc-?[0-9]+".toRegex(), "")
139-
140-
val subjectFile = e.distributionFile("vote-email-subject.txt").relativeTo(projectDir)
141-
val bodyFile = e.distributionFile("vote-email-body.txt").relativeTo(projectDir)
142-
143-
val emailSubject = "[VOTE] Release $asfProjectName $version"
144-
subjectFile.writeText(emailSubject)
145-
146-
val emailBody =
147-
"""
148-
Hi everyone,
149-
150-
I propose that we release the following RC as the official
151-
$asfProjectName $versionNoRc release.
152-
153-
* This corresponds to the tag: apache-$asfName-$version
154-
* https://github.com/apache/$asfName/commits/apache-$asfName-$version
155-
* https://github.com/apache/$asfName/tree/$gitCommitId
156-
157-
The release tarball, signature, and checksums are here:
158-
* https://dist.apache.org/repos/dist/dev/incubator/$asfName/apache-$asfName-$version
159-
160-
You can find the KEYS file here:
161-
* https://downloads.apache.org/incubator/$asfName/KEYS
162-
163-
Convenience binary artifacts are staged on Nexus. The Maven repository URL is:
164-
* $staginRepoUrl
165-
166-
Please download, verify, and test.
167-
168-
Please vote in the next 72 hours.
169-
170-
[ ] +1 Release this as Apache $asfName $version
171-
[ ] +0
172-
[ ] -1 Do not release this because...
173-
174-
Only PPMC members and mentors have binding votes, but other community members are
175-
encouraged to cast non-binding votes. This vote will pass if there are
176-
3 binding +1 votes and more binding +1 votes than -1 votes.
177-
178-
NB: if this vote pass, a new vote has to be started on the Incubator general mailing
179-
list.
180-
181-
Thanks
182-
Regards
183-
"""
184-
185-
logger.lifecycle(
186-
"""
187-
188-
189-
The email for your release vote mail:
190-
-------------------------------------
191-
192-
Suggested subject: (also in file $subjectFile)
193-
194-
$emailSubject
195-
196-
Suggested body: (also in file $bodyFile)
197-
198-
$emailBody
199-
200-
"""
201-
.trimIndent()
202-
)
203-
bodyFile.writeText(emailBody.trimIndent())
204-
}
205-
}
206-
207-
if (isRelease) {
208-
sourceTarball.configure { finalizedBy(releaseEmailTemplate) }
209-
}
210-
211-
afterEvaluate {
212-
tasks.named("closeApacheStagingRepository") { mustRunAfter(releaseEmailTemplate) }
213-
}
21496
}

‎build.gradle.kts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -122,6 +122,9 @@ tasks.named<RatTask>("rat").configure {
122122
excludes.add("client/python/.openapi-generator/**")
123123
excludes.add("regtests/output/**")
124124

125+
excludes.add("releases/test/bats/**")
126+
excludes.add("releases/test/test_helper/**")
127+
125128
excludes.add("**/*.ipynb")
126129
excludes.add("**/*.iml")
127130
excludes.add("**/*.iws")

‎releases/README.md‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# TODOs
21+
22+
* Allow draft-release.sh to publish to Apache
23+
* Close staging repo when RC is abandoned
24+
* Adopt website (top bar menu, maintained releases, left site menu)
25+
26+
# Apache Polaris Release Infrastructure
27+
28+
This directory holds all the infrastructure parts that are required to draft a Polaris release (RC) and to eventually
29+
release it.
30+
31+
Releases can be created from any Git commit, version branches are supported.
32+
33+
Generally, releases at the ASF follow the following workflow:
34+
35+
1. Draft a release
36+
2. Start a VOTE on the dev mailing list
37+
3. If the VOTE fails, the release has failed - "go to step 1"
38+
4. If the VOTE passes, publish the release
39+
40+
The Polaris project decided to run all release related via GitHub actions - a release is never drafted nor actually
41+
released from a developer machine.
42+
43+
## Technical process
44+
45+
In the `release/bin/` directory are the scripts that are required to draft a release and to publish it as "GA". All
46+
scripts can be called with the `--help` argument to get some usage information. There is also a `--dry-run` option
47+
to inspect what _would_ happen.
48+
49+
The technical process for a release follows the workflow above:
50+
51+
1. `releases/bin/draft-release.sh --major <major-version-number> --minor <minor-version-number> --commit <Git-commit-ID>`
52+
creates a release-candidate. The `--commit` argument is optional and defaults to the HEAD of the local Git
53+
worktree. The patch version number and RC-number are generated automatically. This means, that RC-numbers are
54+
automatically incremented as long as there is no "final" release tag. In that case, the patch version number
55+
is incremented and the RC-number is set to 1.
56+
57+
The Git tag name that will be used follows the pattern `polaris-<major>.<minor>.<patch>-RC<rc-number>`
58+
59+
The content of the `version.txt` file is set to the full release version, for example `1.2.3`.
60+
61+
Gradle will run with the arguments `-Prelease publishToApache closeApacheStagingRepository sourceTarball`.
62+
Both the release artifacts (jars) and the source tarball are signed. The signing and Nexus credentials must be
63+
provided externally using the `ORG_GRADLE_PROJECT_*` environment variables. In dry-run mode, this step runs
64+
Gradle with the arguments `-PjarWithGitInfo -PsignArtifacts -PuseGpgAgent publishToMavenLocal sourceTarball publishToMavenLocal`.
65+
66+
The staging repository ID, which is needed to release the staging repository, and URL are extracted from the
67+
Gradle output and memoized in the files `releases/current-release-staging-repository-id` and
68+
`releases/current-release-staging-repository-url`. The source Git commit ID is memoized in the file
69+
`releases/current-release-commit-id`.
70+
71+
The source tarball will be uploaded to the Apache infrastructure.
72+
73+
Release notes will be generated and included in the file `site/content/in-dev/unreleasd/release-notes.md`. This
74+
makes the release notes available later on the project website within the versioned docs.
75+
76+
The suggested release VOTE email subject and body with the correct links and information are provided.
77+
78+
The last step is to push the Git tag with a Git commit containing the above file changes.
79+
2. Once the release VOTE passed: `releases/bin/publish-release.sh --major <major-version-number> --minor <minor-version-number>`.
80+
81+
The command will find the latest RC tag for the latest patch release of the given major/minor version.
82+
83+
The final Git tag name that will be used follows the pattern `polaris-<major>.<minor>.<patch>` and created from
84+
the latest RC-tag for that version.
85+
86+
The Git tag is then pushed.
87+
88+
Documentation pages for the release will then be copied from the `site/content/in-dev/unreleased` into the
89+
`site/content/releases/<major>.<minor>.<patch>` folder within the `versioned-docs` branch.
90+
The changes for the updated `versioned-docs` branch are then pushed to Git.
91+
92+
The suggested ANNOUNCEMENT email subject and body are provided.
93+
94+
Creating a release in GitHub is the last step.
95+
96+
## Technical requirements
97+
98+
To use the scripts in the `releases/bin/` folder, it is required to have a _full_ clone with all tags and branches.
99+
Shallow clones, which is the default when checking out a Git repository in GitHub actions, will _not_ work properly!
100+
101+
On top, all scripts need privileges to be able to push tags and branches to the GitHub repository - this is an
102+
essential requirement for the scripts to do their job.
103+
104+
The `draft-release.sh` and `publish-release.sh` scripts also require the following environment variables providing
105+
the necessary secrets:
106+
107+
* `ORG_GRADLE_PROJECT_signingKey`
108+
* `ORG_GRADLE_PROJECT_signingPassword`
109+
* `ORG_GRADLE_PROJECT_sonatypeUsername`
110+
* `ORG_GRADLE_PROJECT_sonatypePassword`
111+
112+
GitHub actions running the scripts must provide those secrets and privileges.
113+
114+
## Version branches
115+
116+
The Polaris project may use maintenance version branches following the pattern `release/<major>.x` and
117+
`release/<major>.<minor>`. The scripts mentioned above are already support version branches and have validations for
118+
this use case. Using version branches is not a requirement for the release scripts.
119+
120+
The two scripts `releases/bin/create-release-branch.sh` plus the informative `releases/bin/list-release-branches.sh`
121+
are there to help with version branches. The former must be invoked on the main branch and creates a new major-version
122+
branch using the `release/<major>.x` pattern. The latter must be invoked on a major-version branch and creates a new
123+
minor version branch using the `release/<major>.<minor>` pattern.

0 commit comments

Comments
 (0)